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ABSTRACT 



The purpose of the study was to provide *'baseline"data for deter- 
mining the feasibility of further investigation into the use of quanti- 
tive judgmental data in evaluating school programs for determining program 
budget allocations. The specific objectives were to; 

1. Apply a Cost-Utility Model to a *'real world" situation 
in a public secondary school. 

2. Develop a technique for obtaining dollar values v/hich 
represent program threshold cost. 

3. Measure the judgmental gap - the difference between dollars 
actually budgeted to competing programs and dollars recommended 
to be budgeted based on the output generated from the cost- 
utility analysis. 

CONCLUSION 

The process of evaluating programs for purposes of resource allocation 
provided a transparent pi»ocess for coming to some concensus about the 
relative worth or value of these educational programs. 

The Cost-Utility Analysis procedure stimulated the collection of 
program cost data not previously available to the budget administrator 
such as program threshold cost, which previously was unavailable although 
the project school district has had an operable PPBS accounting system 
for several years . 

The discrepancies between the recommended and actual program budget 
allocations were small enough to justify a more comprehensive application 
of the program evaluation and budget allocation process. 
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CHAPTER 1 



INTRODUCTION 



PROBLKH 

A vast majority of the educational institutiono acroGo the nation 
arc experiencing financial difficulties, Th«ro cxictG a severe liriitaticn 
on the supply of public funds for education, while the demand for public 
expenditure for education appears virtually unlimited. Con'^oquently , 
careful consideration must be given to the budgeting process. Cost- 
benefit and cost effectiveness techniques applied to educational expend- 
itures have been suggested as useful methodologies in optimally allocating 
limited resources to maximize the return from the educational investment. 

Levin and Shank (1970) describe co:5t-bonef it analysis as *'3 measure- 
ment technique in which the total costs of a given program are compared 
with the probable total benefits*" This technique in practice relies 
almost exclusively on quantifiabJ^, tangible events, which limits benefits 
to be measured primarily in terms of cognitive achievement. In addition 
inputs and outputs are reduced to quantification in monetary terms. 
Cofit effectiveness analysis is a variation of cost-benefit analysis with 
the except^' on that there is less emphasis on estimating dollar return 
on the educational resource investment and more on correlating alternative 
monetary allocation to achievement of specified goals. In this sense 
cost-effectiveness i-^etuins output in raw form without converting it to 
a dollar value. The results of this approach applied to education are 
often broad generalizations which arc difficult to translate to specific 
programs at individual schools. 

However appealing the above approaches may be theoretically, Kaufman 
(1967) has observed that "before cost-effectiveness and cost-benefit 
analysis can be used effectively, considerable refinement must be done 
with respect to the relationship between economic concepts and theory 
and the institutional (e.g., human, political and social patterns of 
behavior) framework surrounding education." Levitan (1969) has more 
than adequately demonstrated th^ real vorld considerations of Kaufman's 
observation by citing that while the »^^ob Corp Program could be fully 
justified on equitable and moral grounds, the program was terribly 
vulnerable to profit and loss analysis. One might expect this type of 
vulnerability to occur frequently when considering expenditures for 
educational programs. A legitimate explanation is that both cost- 
benefit and cost-effectiveness analysis fail to consider explicitly 
the decision maker's judgment of program worth or value. That is, 
cost-benefit and cost-effectiveness models do not consider intangible, 
subjective feelings and hunches - both models rely on objective inputs, 
consequently there is a judgmental gap between the output of these models 
and the real world. Raiffa (1970) states, "This judgmental gap is often 
so wide that the analysis does not pass the threshold of relevance, 
consequently the analysis may be (and all too often is) general." 
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This phenomena may be depicted as follows: 
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Tuscher (1971) in an attempt to reduce the judgmental gap betveen 
model output and the real world developed a cost-utility model for optimally 
allocating resources among competing educational programs. Cost-utility 
analysis is quite similar to cost-effectiveness analysis with the major 
difference in that cost-utility analysis can consider both quantitative 
and qualitative determinants of program worth. That is, the decision 
leaker takes into explicit consideration judgments of value v;hen evaluating 
program utility. 

This study is an empirical extension of research completed within 
the past two and one-half years in the area of cost-utility analysis. 
The result of which was the development of a mathematical model to 
evaluate education programs and allocate resources to competing programs 
based on the judgmental determination of program worth or value (utility). 

This study as such is intended to provide "base line*^ data for further 
investigation into the use of judgment in the decision making process 
of allocating a constrained budget to competing educational programs. 
Appendix A provides the rationale and assumptions underlying the develop- 
ment of the cost-utility model. 



OBJECTIVES 



1. To apply the (Tuscher) Cost-Utility Model to a real world 
situation in a public secondary school. 

2. To develop a technique for obtaining dollar values which 
represent program threshold costs. (Threshold cost is 
minimum dollar expenditure necessary to operate a program). 

3. To measure the judgmental gap (the difference between dollars 
actually budgeted to competing programs and dollars recommended 
to be budgeted based upon the cost-utility model). 



BUDGETIiN'G and RESOURCE ALLOCATION 



"The quality of management exhibited in school districts throughout 
the United States is very uneven in terms of technical capabilities 
. . . One of the nost common failure areas center around budgeting 
and financial administration.""^ 



■^Merlin C. Duncan, "An Assessment of Accountability: The State 
of tha Art." Educanional Technology ^ Vol. 11 (January, 1971), P. 27. 



Educational ddntinistrators, however, do not deserve full castignticn 
for these failures. Granted that while come adminiGtratora have exhibited 
poor judgment in the allocation of scarce institutional resources, econ- 
omists, social scientists and operations analysts have in iDore cases than 
not failed to present relevant fTiOdels amenable to the resolution of the 
evaluation and budgeting; problems experienced by the practicing admin- 
istrator. 

Kraft fl969) states: There is no great shortage of theoretical 
analysis in the field of cost-benefit and cost-effect ivene^^s analysis, 
but that relevant empirical material is practically non-existent . . . 
the result is that, it has been impossible to test numerical hypotheses 
at the lovel of the cchool. 

Charles Hitch (1967), Secretary McNamara*G Secretary for Systems 
Analysis comments that: 

Economic analysis is concerned with the allocation of 
resources. Its b£4sic tenet is to maximize the value the 
resources used. In business this reduces itself to maxim- 
izing profits, because both income and outgo are measured 
in the same units (dollars). In defense, and, generally, 
in the public ::;ector we lack a common valuation for objectives 
and rasourci^s ^^' The result is that we have to use one of two 
weaker maxims; Maximize objectives for given resources, or 
minimize resources for given objectives. 

Optimization decision-making theory hos been used extensively by 
operations analysts in the development of r-osourcc allocation models. 
The concept is intuitively quite simple: Giver a mix of resources, 
allocate these resources to maximize some index which reflects the utility 
of some set of goals or objectives. Luce and Raiffa (1937) state this 
concept Jiore formally as: 

Let X be a generic act in a given sot F of feasible 
sets and l<tt f(x) be a index associated with f appraising) 
X; then find X in ? which yields the maximum (or minimum) 
index - i^c., f(x) is greater than or equal to f(x) for 
all X. 

Utilizing optimisation decision theory concepts, cost-utility analysis 
assumes either of two general formats: 

1. For a specified level of utility the dec is ion- maker seeks 
to choose from the sot of T possible resource alternatives 
that mix which optimizes the given index of cost, f Cx) , i*e., 
which minimizes program costs for a specified level of 
utility, or 

2. For a specified level of costs the decisrion-maker seeks to choose 
from the set F of possible resource alternatives that mix which 
optimizes the index of utility, f(x), i.e. which maximizes the 
utility for some fixed level of costs. ^ 



(Emphasis added). 



Ccst-effectivcnoss, coGt*bcndfit and cdist-utility methods havo 
been suggested as a ha^i^ for decl^ion-inaking in education by countless 
authors in the past d^^cade. A comprehensive analysis of cost- effect ivenesis 
methods by Tenkin (1570) reveals that: 

rirm theoretical ba^^es for thcac studies are lacking. 
Theory in the domain of decision-inaking, should provide 
not only a basis for description and explanation, but 
explicit statements of assuriptions underlying the proposed 
rationale or methodology. 

Cost-benefit and cost-utility methods have fared no better in 
providing the educational decision maker with explicit statements of 
assumptions upon which the resource allocation model is based. 

The educational decision-maker, when optimizing the allocation of 
resources to programs, is faced with alternative decision sets with 
respect to a proxmiatc measure of the relationship between educational 
inputs and outputs. Considering the present state of the art, he may 
choose to characterise this relationship as one of cost versus indirect 
measurement of the attainment of program objectives, such as pupil- 
teacher ratios, dollar expenditure per pupil, etc., or he may decide 
to identify this relationship as one of costs versus some comprehensive 
measure of the attainment of program objectives, such as an approximate 
measure of program worth or value (utility). 

Robert N. Anthony (1965) in Planning and Control Systems; A Framework 
for Analysis writes that every budgetary system comprises three processes, 
and that while these processes are not always distinguishable operationally 
they may be identified as follows; 

Planning involves the detemination of objectives, the 
evaluation of actions and the authorization of select programs. 

Hanac ement involves the programming of appx^oved goals into 
specific programs, the design of organizational units to carry 
out approved programs and the staffing of these units and the 
procurement of necessary resources. 

Control refers to the process of binding operating officials 
to the policies and plans set forth by their superiors. 

Schick (1956) utilizing Anthony*s conceptual framework in a longi- 
tudinal study of budgetary reform in the Federal Government observed 
that since 1920 these budgetary processes have singulary dominated budgetary 
practices. Schick notes that the control process predominated in the 
years from 1920 until 1935. Here the primary concern was to develop 
systems for expenditure control. Consequently, there followed a demand 
for trained accountants. The management process was dominate from 1935 
until the early 1960 's. During this era the management process received 
prime consideration along with the advent of performance contracting in 
governmental agencies - an interesting concept which has received con- 
siderable attention in American elementary and secondary education. 



"Performance Contracting," Phi Delta Kappan , Vol. 52, No. 7 (1971). 



In the third and present era, the planning era, we see a major emphasis 
in the integration of budgeting and planning as typified by the introduction 
of PPB in all agencies of the federal government. The planning stage 
has been influenced by the folJ owing trends, Schick (1966): 

a. Economic analysis has had an increasing role in shaping 
fiscal and budgetary policy. 

b. Hew informational and decision-making techniques have 
enlarged the applicability of objective analysis to 
policy making. 

c. A convergence has gradually occurred between the overall 
planning process and the budgetary process. 

Kopff (1970) relates that the first trend, especially microanalysis, 
has- been the major thrust responsible for cost-utility analysis. This 
analysis developed initially in efforts by welfare economists to generate 
a science of public finance. Kopff (1970, p. 72) states: 

Such a science of public finance, predicated on the principle 
of marginal utility, would furnish objective criteria for determ- 
ining the optimal allocation of public funds amongst competing 
uses by appraising the marginal costs and the marginal benefits 
that would accrue from alternatives, thereby determining the 
combination which maximized utility. 

The second trend, new informational and decision-making techniques, is 
exemplified by the electronic computer in what Helvey (1971) describes 
as The Age of Information , and the application of comparatively recent 
developments in statistical decision theory such as Fishburn's (196U, 
1970) Value and Decision Theory and Utility for Decision Making and 
Chernoff's (1959) Elementary Decision Theory . 

In his dissertation, Charles T. Nephew (1969), examined the fiscal 
allocation patterns of 119 school districts across ten states to determine 
and specify any functional relationship between resource inputs and 
educational outputs (quality). Nephew (1969, p. 61) concludes from his 
study that "money and the manner in which it is allocated to expenditure 
sub-categories does have an effect on school output even after the 
effect of socio-economic background has been accounted for.'* He further* 
states : 

. One can reasonably conclude that local school boards and 
their staffs must assign priorities to school objectives before 
allocating the available financial resources to specific categories 
within the school budget . * . across the board cuts or increases 
are not the answer. 

The setting of priorities for budgeting to competing educational 
programs is not entirely an objective process. Quite the contrary, it 
is primarily a judgmental process. 



LEVELS or P^SOURCt ALLOCATION 

The budgeting process for a local educational agency may be concept- 
ualized as a procesc of resource allocation reallocation progroscing 
from the tnacro-crganizational level to the nicro-orjjanisaticnal level 
(see fig, 1.0). 

The budget conctraint at the district (macro) level is primarily 
determined by political events at the local, st,atc and federal levels. 
These political events would be exemplified at the local level through 
bond issues, at the state level through subsidies and the fc-deral {govern- 
ment through federal revenue- sharing. 

The process by v/hich the freely allocatablo portion the K dollar 
budget (see fig. 1.0) is allocated to individual buildings or schools 
is not of prirrary concern for this study, but only as this process provides 
a constrained budget for the individual school. It- should suffice to 
say that this can be done in most cases simply on a per pupil dollar 
allocation formula. 

At the third level we are given a constrained budget of k . , dollars 
to be distributed in sorie manner to j programs within school i*^ The basic 
questions to be answered at this level is "what do we value in education" 
and how should we allocate the constrained budget so as to maximize 
some measure of total program worth, i.e., how shoul'd we budget the 
k£j dollars to the competing programs within school i to maximize the 
total worth (utility) of the j programs? 

The allocation of funds to specific programs by the process suggested 
for level three budgeting provides a budget constraint of dollars 
for each of the j programs. At this level we are not so concerned with 
the value (utility) of a program per so, but givon a constrained budget 
of k... dollars for program j, what is the most "effective" mix of resources 
or the "best" alternative solution to satisfy that program's objectives. 

School personnel are continually faced with situations which involve 
choosing a candidate solution from competing alternatives. Choosing 
among competing alternatives is essentially a decision process. The nature 
of the decision process implies a futuristic orientation, one of anti- 
cipating consequences of acting on "^he decision alternatives. Since the 
educational decision maker must and does make a final choice, it se^ms 
apparent that in some way the administrator or teacher must weigh the 
anticipated consequences against som:? factors (criteria) to reach a final 
overriding value assessment that determines his choice. The argument is 
of course that the decision process involves a high degree of uncertainty 
and that values and personal judgment play an important part in arriving 
at an educational budget allocation decision. 

A. LEVEL 3 BUDGETING PROCESS (MODEL I) 

Model I is referenced from the basic value decision model (see fig. 
2.0). In Model I the a^ are programs such as math, science, English, 
social studies, etc. in whicli the utility of program i is the sum of 
weighted utilities across the k criteria. That is = E ^•i.Wj^ • 
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• The procecs of allocating expenditures to educational program!; to 
myMxze the* ooripo^^ite totiil utility (total profiram worth) can begin once 
the prografliss havo been ^^^almtcd and the constrained budg<»t and individual 
program thret^hold corts have h^t^n detemincd. An interactive conputer 
proerarn has been urittiin vli^^■h. cm be used by the d<^^ision isak^r to eval- 
uate progran*^ at.: alioiia^--* '^^npirnditurig;; to profir<trT^ which will meKimise 
total prcgian v/o: }:. The alj^^orlthirj to ruaxLt:iize total pragraa worth is 
based on optimiuctlon theory using the Lagrange Muitiplifsr Technique 
and recuralon KJt^thodts. 

In order to allocate limited resources among conpetlr^g programs bailed 
on the utility of these programs;, it io necessary fir^t to c^tablit/.h a 
functional relation:>hip betvecn the utility of a program and the cost of 
each program at variouc> level:^ of cost. The utility of a progru?; is 
defined a^ a real vaiui-d n;:mber which expresses an evaluator's judgr.cvnt 
of the degree to which a given program has the potential to contribute 
to the satisfaction of ^omo stated criterion or objective. It is assumed 
that tho evaluator's judgment or preference takes into conaideraticn both 
qvialitative and quantitative factors* The amount of money or resources 
allocated to a given i>i ■ am will depend on the degree to which that program 
catiixfi^c a given set' of criteria or objectives* It is observed fix)m 
the above discu.'jcion that utility i^ a function of level of expenditure 
and come t^et of crlt«.*ria or objectives, 

A number of relationships between utility and level of expenditure 
become iRunediately obvious, which will enable us to determine a relatively 
good approximation of the cost-utility curve, and subsequently a functional 
reiaticnchip botw^^en program expenditure and program utility. These 
rclationrhipr: or undt-rlylng aissumptions are: 

1. /: program allocated zero resources has zero utility for any 
cct of criteria or objectives* 

2. The utility of a progt^am increases as it's level of expenditure 

incrcat;c*St 

3. There 3c? a minimum level of program expenditure (threshold level) 
whicl) mm allocated to a given program if it is to operate 

at till* Corresponding to this threshold level of program expend- 
iture is a minimum level of acceptable utility. A utility value 
b^,^low this level implies a non-operable level of program expend- 
iture*. 

u. The utility of a program does not increase indefinitely as it 
is allocated greater and greater expenditure levels. This seems 
to be a rctisonable assumption. For example, it is very doubt- 
ful th^t if teachers salaries were doubled, the value or worth 
(utility) of the p^:^5gram vvould also double. This is, beyond 
the threshold l*^vtd of utility the marginal utility decreases as 
thr? expenditure Icivel increases. In an economic context this 
astiumption means that fjmall allocations lead to essentially 
c-cro returns, while largo onen have a saturation effect, the 
"law of diminishing returns/* 
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RESOURCE ALLOCATION (MODEL 1) 



The results of the previous sections have provided a functional 
relationship between the utility, U^Cx^), and the level of expenditure, 
Xi, for any program i5i-l52j. . ."L, That is, given Program Threshold 
cost, C.,i=l,2, . . , L, and the corresponding values U^(x^), i = 1, 
2, • . L derived by methods outlined in the evaluation model, it is 
assumed that U.(x^) and are functionally related by the logrithmic 
function: 

-Bi/x. where 6- = 2C. 

Ui(x.) = U.e 

The problem now is to find the values x^, i = 1, 2, . . . L. constrained 
by N dollars allocated amongst L pr^ograms such that 
and the sum, is maximized, i.e., L 

L E = N 

I U.(xj_), i=l 

i=l ^ 

L 

MAX: E U.(x.) (1) 
i=l ' ' 



S.T.: N (2) 

For example, consider the case where L = 2, By the method of 
Lagrange multipliers, a general method for finding an extreme value 
(maximum or minimum) of a function subject to one or more constraints, 

2 -3./ 2 
F(x^, X2,X) ^ I U.e ^^^i+X( Z x. - N). 
i=l i=l 

Sample output from the expenditure allocation algorithm would appear as 
follows : 



Program 


Expenditure 


Utility 


Math 


$10,200 


18.7 


Science 


9,600 


16.4 


English 


8,300 


9.3 


Social Studies 


8,700 


10.6 


Total 


$37,000 


55.0 



LEVEL k BUDGETING PROCESS (MODEL II) 



After the budget constraints for programs within a school are 
determined, the decision maker is faced with choosing the ^^best^^ mix 
of alternative resources consistent with some identified decision criteria. 

In this situation we are concerned with identifying an objective 
function which combines information concerning the utility of outcomes 
(see fig. 3.0) and the probability of outcomes (see fig. 4.0) into an 
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estir-ata oc e>:pact3d utility ^ ass'^ninj ths daciulcTi role is to t?,ai<iniizo 
expected utility* 
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The objective function then becomes: 
U. = U u(C.)., f(C.)j 

where Uj = expected utility associated with alternative a^ 
U = functional notation 
U(C£)j set of utility estimates 

f(C^)j»= probability density function of criteria estimates 

let U^j = the expected utility of given aj 

If is measured on a continuous scale then 

"ij = U(Ci). f(C.)j dC. 

and then assuming that the criteria are valuewise independent the expected 
utility of alternative aj becomes: 

U. = E /.:u(Ci). fiC,)^ dC._ 

The decision r^ule is then to choose that alternative which has the 
greatest expected utility. Model II is related only to a general 
scheme for allocating educational resources, and is not an intr^gal 
part of this investigation. 
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I. DESCRIPTION OF ACTIVITIES 



This research investigation wa3 undertahea in cooperation with 
administrative staff and teaching pei^sonnei at Pennsbury High School. 
The Pennsbury School District, located just outside Philadelphia, was 
one of six school districts in the urate of Pennsylvania selected to 
participate with the Pels Institute of Local and State Government in tha 
development of an Educational PPB System (1965). Consequently, the 
Pennsbury School District is currently operating with a program accounting 
system under which each school maintains some autonomy for allocating 
instructional expenditures by programs. The research activities reported 
herein were conducted at Pennsbury Higli ochooi utilizing an approach 
referred to as Cost-Utility Analysis:. Appendix A delineates the rationale 
and underlying assumptions inherent in the procedui-^es as de:;cribe'd in 
this chapter. 

The Cost-Utility Model required that programs competing for resource 
expenditures be specified along with a set of decision criteria by v/hich 
the relative value or worth of each program was judged by ecich of the 
decision-makers. Subsequently nuTierical measures (utilities.) were assigned 
to each of the cells In the program - criteria matrix. That is, the 
decision-makers assigned values to program-criteria combinations. The 
values assigned represented the potential vciue or worth of each program 
with respect to satisfying the specified program criteria (see fig. 5.0) 




Criteria 



Pig. 5.0 
12 



In addition to the above requirements the decision-makers were 
required to assign relative weights to the criteria identified in the 
program-criteria matrix. 

The numerical values assigned to the cells in the program-criteria 
matrix were checked for consistency with the decision-maker's judgments 
by using the Churchman-Ackof f approximate measure of value or worth 
(see Appendix A, pages A7 - A9). 

The activities described above concluded the first part on the 
program evaluation part of the budgeting process. Before the expenditure 
allocation process began the constrained budget, program constrained 
maximum and threshold cost of each program were determined. The con- 
strained budget is the dollar expenditure allocation given to Pennsbury 
High School by the district office to be allocated among the competing 
educational programs operating at Pennsbury High School. The program 
threshold cost is the minimum dollar expenditure required to maintain 
the current operating program at a status quo level. Any expenditure 
greater than the threshold cost would mean that the program is exper- 
iencing growth in a value-added sense. The program constrained maximum 
is the dollar value above which a specified program cannot effectively 
utilize its resources. 

The process of allocating expenditures to educational programs to 
maximize the composite total utility (total program worth) began after 
the programs had been evaluated and the constrained budget and individual 
program program threshold costs had been determined. An interactive 
computer program was utilized to aggregate evaluation data and generate 
allocation expenditures to programs. The algorithm to maximize total 
program worth is based on optimization theory using the Lagrange Multiplier 
Method and recursion met hods developed by Newton (for a complete explan- 
ation see Appendix A, pages AlO - A13). 

II. PROCEDURES 

Step 1 - Meet and Discuss Operational Plans 

The project director met with administrative personnel of 
Pennsbury High School during late September, 1972, to discuss and schedule 
the research activities for the ensuing year. The research activities 
and time schedule for completing these activities were coordinated to 
interface with the budget preparation calendar of the Pennsbury School 
District (see Appendix B for activities and time schedule). 



Step 2 - Determine Programs 

Programs identified for inclusion in the investigation were selected 
on the basis of the following practical criteria: 

1. Total instructional expenditures in the sub-categories 
of textbooks 5 supplies and audio/visual aids for all 
programs selected should account for at least 75% of 
total program expenditure as determined from past 
program expenditure allocations. 

ERLC 
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2. Only programs whose chairman has agreed to participate 
will be included in study. 

Consistent with the above decision criteria the following programs were 
selected: 

1. Business Education 4. Mathematics 

2. English 5. Social Studies 

3. Language 5. Science 

Step 3 - Identify DeciGion-Makers 

The decision-makers in this study are defined to be: 

1. Curriculum chairmen of programs selected for study. 

2. Building budget administrator. 

Since the curriculum chairmen are in reality competing for a ^^fair share" 
of the constrained budget j it is reasonable to assume that their judgment 
may be biased in favor of their own program. To adjust for the possible 
bias each program was represented by a single decision-maker, the program 
chairman. The building budget administrator was included as a decision- 
maker because he had the responsibility and authority for resource 
allocation decisions at the building level. 



SteD H - Determine Criteria 



The decision-makers met, discussed" and defined the criteria upon 
which the program allocation decisions should be based. The relative 
worth or value of a program was determined by the degree to which each 
program was judged to have the potential to contribute to the satisfaction 
of these predetermined criteria. Each program was then budgeted monies 
according to its aggregated program worth. The following criteria and 
accompanying definitions represent a consensus of the decision-makers 
of those considerations deemed to be important in the program budget 
allocation decision: 



Relevance The degree to which a program satisfied the social 

cultural goals and objectives of the institution. 

Generality The degree to which the outputs of a given program 
are utilized by the other programs. 

Assimilation The degree to which a program utilizes the outputs of 
all other programs . 

Urgency The degree to which an evaluator wishes to give 

explicit recognition to pressure groups in the 
budgeting process with respect to a given program. 

Need The degi-^ee to which an evaluator wishes to emphasize 

a higher priority because of special circumstances. 
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Step 5 - Assigning Relative Weights to Crilaria 



The basis for assigning weights to each of the criteria rests with 
the assumption that the criteria are considered to vary in importance 
as they relate to budget allocation decision. Each decision maker 
weighted the criteria according to the following procedures: 

1. Rank order the criteria 

2. Assign a number between 0 and 100 to each criterion, so 
that these numbers are consistent with your rank order. 
The sum total not to exceed 100. 

3. Check for inconsistencies between your judgments and the 
numerical representation of your judgment using Churchman- 
Ackoff Approximate Measure of Value (see Appendix A). 



RANK ' CRITERIA IMPORTANCE 



1 



3 



5 



EXAMPLE : 

1. Rank Order Need ^0 2. Assign Values Consistent 

Cr»iteria Generality 30 with rank order 

Relevance 15 

Assimilation 10 3. Check for inconsistency 

Urgency 5 

The criteria weights were determined by averaging the importance scores 
over all evaluators for each criteria and then normalizing the averaged 
scores. The procedure utilized by the decision makers is detailed 
in Appendix C. 

Step 6 - Determine Budget Constraint, 

Program thi^eshold costs and program constrained maximums. 

The budget constraint is the dollar expenditure allocated to 
Pennsbury High School by the district office to be reallocated among 
the six competing educational programs within the instructional sub- 



id 
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category, textbooks • " 



Program threshold cost is the minimum dollar expenditure required to 
maintain the current program at the status quo level. Any expenditure 
greater than the threshold cost would mean that the program is exper- 
iencing growth in a value-added sense. 



a 



Program threshold costs were determined by first examining budget 
allocation patterns and approved program budget requests in the text- 
book category during the two previous years (see Appendix D). This 
historical data provided a baseline from which to zero in on the projected 
program threshold costs required for this study. 

By examining the approved budget requests during the previous year, 
the budget administrator was able to identify for each program the 
approved list of textbooks and resource materials. Working from this 
approved list of textbook and resource materials, the program threshold 
costs were determined by summing the cost incurred for textbook and 
materials replacement and the cost for obtaining additional textbooks 
and materials based on any increase in projected program enrollment. 
The threshold dollar figure is a guaranteed or minimum level of expend- 
iture allocated to a given program. 

Program constrained maximum is a dollar value greater than or equal 
to the program threshold cost. It is the dollar value above which a 
specified program cannot effectively utilize its resources. 

This value was subjectively determined by the budget administrator. 
It was determined in part by comparing for each program the previous 
year's approved budget against demands to exceed that budget allocation. 

Step 7 ~ Assigning Numerical Values (Utilities) to Program-Criteria 
Matrix. 

Each decision-maker evaluated all programs against each criteria 
according to the following procedure (see Appendix E): 

Assign values to each cell of the Program-Criterion Matrices on the last 
page according to the following: 

1. Rank order the program alternatives for the first criterion. 

2. Assign a number between 0 and 100 to each program alternative, 
such that these numbers are consistent with 1. 

3. Check for inconsistency between your judgments and the numerical 
representation of your judgments. 
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"The intent of the original investigated was to use all three of 
the instructional sub-categories; however, district budgeting policy 
prohibited the lumping of funds across categories for the resource 
allocation. process . Subsequently, that sub-category which accounted for 
the largest percentage of the total of all three sub-categories was 
used in the investigation. 
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EXAMPLE: 



Relevance 



Rank order 
programs 



English 
Social Studies 
Science 
Mathematics 



90 
75 
70 
60 



Assign Values 
Consistent with 
rank order 



3. Check for inconsistency, if inconsistent, adjust numerical 
values to obtain consistency with your judgment. 

Inconsistency was determined by utilizing the Church-Ackof f procedure 
for obtaining an approximate measure of v;orth or value. A composite 
program-criteria matrix was obtained by averaging score values for each 
cel-l across all decision makers. This composite matrix was used to 
obtain a single measure of program v;orth by summing the weighted averages 
for each program. 

Step 8 - Optimize Budget Allocations 

(see Appendix A, pages AlO - A13) 

Step 9 - Determine Judgmental Gap 

The judgmental gap was operationally defined to be the difference 
between dollars actually budgeted to the six competing programs and the 
dollars recommended to be budgeted based upon the cost-utility model 
in the instructional sub-category - textbook. 
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CHAPTER III 
RESULTS 



The primary purpose of this study was to provide "baseline" data 
for determining the feasibility of further investigation into the use of 
quantifying judgment in the decision leaking process. The specific objectives 
were to : 

1. Apply a Cost-Utility Model to a 'Veal world" situetion 
in a public secondary school. 

2. Develop a technique for obtaining dollar values which 
represent program threshold cost. 

3. Measure the judgmental gap - the difference between dollars 
actually budgeted to coinpeting programs and dollars recommended 
to be budgeted based on the output generated from the cost- 
utility analysis. 

Objective 1 ~ Results 

The data required as input for the computer generated outcomes were 
obtained by the procedures delineated in steps two through eight in 
Chapter II. That is, six prograins, Business Education, English, Language, 
Mathematics, Social Studies and Science were evaluated against the following 
five criteria (see Appendix F): Relevance, Generality, Assimilation, 
Urgengy and Need by each of the seven decision makers. In addition the 
criteria were weighted and the program threshold costs, program constrained 
maximum and budget constraint were determined (see Appendix D). These 
data were fed into the computer with the following results being generated; 



\../^\iT I>3hThT]CTI0'\)S? 
?YES 

THIS PHOGF/^y' ALL.OV.S ^ U- TO '^O Fyj^lJIATHPS F/v^^.M^TF FrnPAiT OM at . PPOP^^'^''? 



O'vj THP: P/^SIS OF PIV/F^vJ rhlTF.Fl^ r\ir op.ifctivfs. np to 1P PiOrP^v^: 
TKF USH'.rt IS THKM ^l.LnV.FP TO V.FIC-HI TMF PIFF^P.FMT ri-.TTFF'T^> 
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?EN)G,SOCST, y/Ni-u I ,^•MSKl•>l.^^^JC• 

,\}r«vE 'iWU CFITEHI^ THENJ T/PE 'f 
? H E I. , # G E M 1 N) E F r > ^ • S ^ ' , n i r 1. 

WILL THE ¥U^V1 EV/^L^JPTOi* Fl.F-'^'.^E STP^J Tv, 
?COH^^ETT 

you V.ILL PE PSKEP TO ASSIGM ^TrMT/ CO TO 100) ^■:/>fVf ppori-Av 

PASED OM EACH OF T !K ChlTE-a^. 



CST 

y-oiH 

SCI 

PUSED 
LAMG 



— ?9r> 

— ? s n 

7 60 



? 5b 

? 55 



EMG 
SOCST 

SCI 

PUSED 
LAMG 



? -)n 

? 7 5 




NEED 



EMH 
SOCST 
X^ATJ-1 
SCI 

PUSED 
LAWG 



— ? w n 

---?75 

? 65 

? 70 

? 55 

?60 



ASS^' 



E.vlG 
SOCST 
MATt^ 
SCI 

BUSED 
LAVG 



?90 

? 5 

? 70 

?80 

---?f.5 
?60 



IJHG 



ENG 

SOCSl 

MATK 

SCI 

PUSED 

LA>JG 



— V8n 

•--?75 

? 65 

?70 

---?55 
? 60 



WAlVT to ZH't.C^ im'i' uriLlTf ASSI CNIN-^EMTS FOH C0\'SI ST^'MCY/ 

USING H-: CHUhC^/jAM-ACK ')FF PROCEDUhE? 

---?M0 
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?iv30 

V. ILL THE MEXl Fv' r^LM-vi o.^. i-LFASF S I ^ \) 1 . 

(TYFF • + • TO TFi^-Tv'^'I ^: THr! F'A/ALM/^^ TI FHOC H'nnb:'- . ) 

?40LLIH.A\3 

YOU WILL BE ASKrD 10 ASSTC-x) ^ riTTLTT/ (0 TO lOO) TO F^C-? Pl-orn^- 
PASEL OA] F.AC-4 OP' T^^K CrlTF^^lA. 



HEL 



SOCST 
SCI 

pnsED 



- - ? i b 
--? 100 

- - ? 0 

- - ? '/ 5 
--?60 



E'>iG 
SOCF-1 

SCI 
nUSED 

L a:\jg 



100 
90 

- - ■? H b 
--?50 
--?70 



'^JEED 



E'.>JG 
SGCST 

iV;AT^^[ 

SCI 

F-JSED 

LA-MC 



?e,o 

? 100 

? 7 0 

? C) 

?90 

? 50 



ASS^^j 



E.\}G 

SOCST 

yjATH 

SCI 

FUSED 

LA:v3G 



? ) 0 

? ) b 

? 6 S 

? 60 

? 100 

- ? 7 o 



UuG 



e:og ---?75 

SOCST ?^0 

f^.ATK ?S5 

SCI — ?ioo 

BUSED ? ^5 

LA\iG V60 



WAiViT TO CHECK rOUH UTILITY A SS I G ^JY'F'MT S FOR OONJ SI £T Lv^nJC f ^ 
USING TKF CHURC'L^a:m-^aC'<OKF FHOCEDMHK? 
?(SJO 
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V.AMT A PiaNTOMT OK YOUr: IM^^virUAl. ?:v'^l/ J^T I OM Y.^Tt-I^* ? 
?W 

V. ILL THE WEXT EUALUATOR PLEASE SIGW I ^3 . 

(TYPE TO TEFi'MIMPTE THE Kv; "^LMAT 1 0\f PP.OCEDMPK • ) 

?PACHMA^O 

YOU WILL BE ASKED TO ASSIGN] A UTILITY (O TO inp) TO F/^C^ PHOGR/^^^] 
PASED ON EACH OF T^^E CHITEHTA* 



REl 



ENG ?100 

SOCST ?90 

iX'/^TH ? 70 

SCI ---TVS 

BUSED ?H0 

L^MG -"-750 

GEN 

Ei>JG ? 1 no 

SOCST ---7 70 

K;ATK 7:^5 

SCI ---?;)n 

HUSED ---IHO 

LANG ---7 60 

>JEED 

E'.nJG 7 1 00 

SOCST 765 

MATK ?90 

SCI ---785 

BUSED -"-770 

LANG -"-7 75 

ASSM 

ENG 7 70 

SOCST ---?H5 

MATH ---7 75 

SCI 7100 

nt)SED -"-7 60 

LA\1G ---7 65 
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E\^G ?90 

SOCST 7 100 

C^ATH ?65 

SCI '•--?^^0 

PUSED ?70 

L^NJG 7 75 

V.A.vT TO CHECK YO"':- UJlLllf A [ c njm E'^T 5 ^ O;^ C0\' S I ST F^JC Y > 
lJ S I v G TK E 1.) r: CH X' >J - A 0 ) F r P i ^ 0 C FPU h >:? 
?'NJO 
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vvILL THE MEX'T El/AiJIATOi- Kr.KASE SIGN) T\i. 

(T/FF: •** TO TEKv,i-sj/^TF. THE EVAI UATION] I FvOCEDn:?E^ ) 

---7V.ILLIMG 



YOU V;iLL PE /^SKED TO ^SSU:V WIW.ITY (H TO jnOD TO ?,^r/^ PROPHAM 
BASED ON EACH OF THli: CHITEBI.^* 



HEL 

ENJG ?100 

SOnST 1)3 

^i^TH ?^^0 

SCI i^rs 

BUSED ?35 

LAiVG ?30 



E^OG VI on 

see ST 7 60 

K'ATH ?90 

SCI VoS 

BUSED ?10 

LANG ;P0 

NEED 

E'MG V60 

SOCST ?70 

T^mTK VbO 

SCI 7 1 on 

BIJSE)? ?-<0 

lalnjc: - -7/iri 



A S S[<' 

E^JG 7-30 

SOCST ?--;n 

v.ATH ?r-^0 

SCI ?ion 

PTJSED ?4n 

LA^JG ?30 



URG 

E'MG ?70 

SOCST 

MATH 'U\0 

SCI ? 1 no 

FUSED ?60 

LA^3G ?5n 



WAMT TO C^ECK YOUH UTILITY A SSI G\]'v^F\TTS FOP CO OST STI^^MCY* 
USIMG THE CHnhCHrl^^'-AC'^OFF PHOCEDUPF? 
— -TNG 
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WANT A PRINTOUT OF YOMi; I M r>i v I DIJ/^^L EVALHATTOM ^(^'U'IaI 

— im 



WILL THE NEXT EVALUATOH PLEASK SI G'>] IM. 

(TYPE TO TERMINATE TME EUALHATION) PROCEDURE.) 

? DOUGHERTY 

YOU WILL PE ASKED TO OS.SIC-^J A UTILIT/ CO TO 100) TO K^C'"^ PrOPR^^' 
EASED ON EACH OF THE CHITEHIA. 



REL 



ENG ?90 

SOCST 

MATH ?-^>0 

SCI ?50 

BUSED ?100 

LANG ?P.5 



GEN 



ENG ?100 

SOCST 

WATH ?9 5 

SCI — ?P0 

BUSED ?15 

LANG 19.5 

NEED 

ENG ?95 

SOCST ?90 

MATH ?80 

SCI 720 

BUSED ?i5 

LANG 125 

ASSM 

ENG ?50 

SOCST ' ?y<0 

MATH ?8 5 

SCI ---?95 

PUSED ?100 

LANG ?P.5 

URG 

ENG ?100 

SOCST ?50 

MATH ?9 5 

SCI ?25 

BUSED ?90 

LANG ?10 

WANT TO CflECK YOUR UTILITY ASSIGNMENTS FOR CONSISTENCY^ 
USING THE CHURCHMAN-ACKOFF PROCEDURE? 
?N0 



WANT A PRINTOUT OF YOUH I '^DIVIDUAL KVALJIatI OM y^TMlv? 
?N0 



WILL THE NEXT EUALTJPTOH PLK'^SE SIGN] IN. 

(TYPE TO TEPI^IMATE THR KVALf JATI ON PROCEDURE, ) 

?COBR 

YOU WILL BE ASKED TO ASSIGM A UTILITY (0 TO 100) TO EACH PFOGP^ 
BASED O^J EACH OF' THE CHITFRIA. 



REL 



ENG 

SOCST 

KjATH 

SCI 

BUSED 

LANG 



100 
--?95 

10 
--?30 
--790 
--?75 



ENG 
SOCST 
MATH 
SCI 

BUSED 
LANG 



1 00 

- - ? 2 0 
10 
--?^0 

30 



MEED 



ENG 
SOCST 
MATH 
SCI 

BUSED 
LANG 



100 
--?/jO 

10 
--7P10 
--?50 
--?30 



ASSKi 



ENG 

SOCST 

MATH 

SCI 

BUSED 

LANG 



? 100 

? 30 

? 1 0 

?20 



?50 

? AO 



URG 



ENG 
SOCST 
MATH 
SCI 

BUSED 
LA;MG 



— ? 1 on 

?40 

? 1 0 

19.0 

?50 

? 30 



I'.ANT TO CHECK YOUl: UTILITY ASSI Givl^iENTS POP CGMSI STF^3CY^ 
USING THE CHURCH>:AN-ACKOr F pKOClvDnpE? 
?N0 



WANT A PHINTOUT OF Y.UII: I^^DIVIL^/^L EU/M.UrTIOrg v:/^•rR^<? 
?M0 



WILL THE MEXT EVALUP.TOP. FLEASE SIG\i IM» 

(TYPE TO TERiYi'MATE TH?: EV.^LliAT I OlvJ PKOCEDURF. ) 

?DUANE 

YOU V:iLL PE ASKED TO (^SSlGN A UTILITY CO TO 100) TO EPCH PPOOI-Av 
RASED 0^ EACH OF THE CF.ITEPIA* 



REL 



ENG 

SOCST 

iMATH 

SCI 

BUSED 

LANG 



? 9 5 



?oO 

? VO 

?60 



EIvG 

SOCST 

MATH 

SCI 

BUSED 

LANG 



? 1 00 

? 9 5 

?90 

?P5 

?80 

? 70 



MEED 



ENG 

SOCST 

K'ATH 

SCI 

FUSED 

LANG 



? 100 

? 7 5 

? 50 

? 9 5 

? 7 0 

---?^-^0 



ASSM 



ENG 
SOCST 
WATH 
SCI 

BUSED 
LANG 



?50 

?60 

---?70 

— ? inr: 

?95 

?70 



URG 



ENG 

SOCST 

L^ATH 

SCI 

BUSED 

LANG 



100 
---?75 

---?90 

?^S0 

---?60 



I'^ANT TO CHECK YOUR UTILITY ASSIGNMENTS FOR COX^ST STEvJCY^ 
USING TI-IE CHURCHMA\J-ACKOFF PROCEDURE? 
?M0 
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ivA.MT A PRINTOUT OF fOV-fi IMF I VI DUAL EVAT.I.JATI 0;^J MATRIX? 
?N0 

WANT THE COMPOSITE EVALUATION ?<ATHIX? 
?YES 



REL GEl>I 'JEEE) AS^--' 

ENG 86.25 90.75 86. 68.75 

SOCST 92.00 76. 73.75 71 . 

MATH 68.75 75.00 58.13 58.13 

SCI 67.50 65.00 70.63 8 1.88 

BUSED 70.00 50.00 6R.50 75.63 

LAMG 51.25 49.38 45.00 e)5.00 

URG 

EMG 89.38 

SOCST 70.00 

MATH 62.50 

SCI 71.88 

BUSED 72.50 

LANG 50.6 3 
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Iv'ANT THE STANDARD DEVI ATI OI'O yj.'Vri-.IX? 
— -?YES 



REL GEi^J NJEED ^SS'M 

MG 29.02 3.31 16.76 r.0.?7 

SOCST 5.9 6 15.40 16.5-;! l-l.S^^ 

MATH 23.55 R5.00 PP.. 90 S5.61 

SCI 17. 50 30. PI 30. 56 P6.9P 

BUSED 19. Pn ?6.5:"'. PI. 51 PP.9 7 

LAMG 15.36 PO.n? 15.21 19.^0 

UHG 

Km 11.S4 
SOCST 17.85 
KATH 25.96 
SCI P9.99 
BUSED 15.41 
LANG 19. A A 

ASSIGN A WEIGHTING FrCTOi" BETI.EF..M 0 AMD 100 TO EACH CniTFRTOX' 
AS THE NAME OF TKF. CH T T I n-vj IP PHUITED OUT. 



REL ---IPH 

GEN ?P3 

MEED ?22 

ASSM ?15 

URG ?1P 



PROGRAM UTILITY 

ENG 92.50 

SOCST 8 1.15 

MATH 76.20 

SCI 88.65 

BUSED 7 7.25 

LANG 60.90 
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E^JTER THE THRESHOLD COST C I » POLLAP?) OF EACH PFOGHAM 
AS THE IS3AME OF THE PHOGhAr<' IS PF^INTEP OUT. 



ElMG 

SOCST 

MATH 

SCI 

BUSED 

LAMG 



— ?sooo 

??000 

72000 

???500 

7RO00 

? ISOO 



EiNjTER THE CONSTK^I A-liiP rs/^AlI^U>: ( 1 r^J DOLLAP.'^) FOP' EAC»^ PPOGP^^ 

AS THE NAHE OF THE PPOGhAX- IS PPl^JTED OUT. 

(A ZERO VALUE V;ILL CAUSE THE PPOGi^/^^l TO ^E 0(<'I TT^D 

FROM THE OPTIMIZ;ATIO>3) 



ENG 75000 

SOCST 7^000 

MATH 73000 

SCI ---75000 

BUSED 7^000 

LANG 7 I POO 



rOUR CO'MSTHAIMED ^;AKIe^l^^^ T. S LEr>S THAM OR EOTJaL TO 
THE THEESKOLD COST FOR THIS PPOGP>^M. 
PLEASE RETYPE THE CO.^JSTRA I '^JFD -jAXI-MJ^'^ FOR LA\TO 
72000 
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WAMT A PLOT OF ONE OR ^lOhE OF THE CnST-HTI LI TV Cnr^v-'S^ 
?YES 

SPECIFY THE PROGRA'MS FOR IvHICH YOU IvAMT THR COST-ttji L I TY CTHtv? 
A.^D THEN TYPE 

?ENG^ SOCST^MATH^ SCI ^ PUSED^ L A\i G* 

ENTER THE MAXIMUM COFT TO BE PLOTTEF. 

(THIS DEFINES THE RIGHT E:\T>-POINT OF THE X-Avj'^.) 

77500 



92.50 + 



80.94 



69.37 



57.81 



^6.25 

UTILITY 
34.69 



1 1 



1 1 1 



1111 

1 1 1 

! 1 PP 

111 PPfcJcA: 
1 1 f^-^cJ.'** 

1 1 6f'v*«*^***^i 
1 *66*P*ic*=*^/J 
**6 PP**i</J/a 

66* 1 p4c>fc3/|/( 
6 6 1 P*t 
12 + 6 112** /J 4 

66 1 ** 44 
66 12*344 
6 11**4 
6 IP* 44 
1 1 . 56 + 6 1 **/i/j 

66 ** 4 
6 1**44 
6 **44 
6 * * * 
0.00 +********4 

+ •.•. + ..••+.... + .... + ,,,, + ,,,,+ ,,,,4.,,,, + ,,,, 

0 COST 



P2P 
* * * 



• • • + 

7500 



SPECIFY THE PliOGHAl^S FOR ViHI CH Y OH va*n}T THF. COST- ' ^ FT UI TY CTJPVK? PLOTTED 
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Objective 2 - Results 

PrograTB threshold cost was defined as the miniTnum dollar expenditure 
required to maintain a program at the current operational level. Any 
expenditure greated than the threshold level would imply that the program 
is experiencing growth in a value-added sense. 

The procedures for obtaining the approximate dollar value representing 
program threshold levels are discussed under Procedures - Step 6 in 
Chapter II. 

Objective 3 - Resjilts 

Objective three was concerned with measuring the discrepancy between 
dollars actually budgeted to competing programs and dollars reocmmended 
to be budgeted based on the output generated from the cost-utility 
analysis (see Table 1.0). 



Program Budget Discrepancy 



Program 



Recommended 
(Dollars) 



Actual 
(Dollars) 



Discrepancy 
(Dollars) 



English 
Social Studies 
Mathematics 
Science 

Business Education 
Language 

Totals 



4,634.17 
4,000.00 
3,000.00 
4,456.57 
3,909.25 
2,000.00 



4,500.00 
4,000.00 
2,500.00 
4,500.00 
3,500.00 
1,500.00 



22,000.00 20,500.00 



134.17 

0 

500.00 
43.43 
409.25 
500.00 



Table 1.0 



The budget constraint utilized in the cost-utility analysis was $22,000.00, 
while the budget constraint for actual expenditures wes $20,500.00. 
This discrepancy of $1,500.00 evolved as a result of the budget administrator 
allocating $1,500 from the $22,000 to a contingency category after the 
program budgets had been determined using the cost-utility approach. 
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CHAPTER IV 



Summary and Conclusions 



Summary 

The study reported herein, as previously indicated, was experimental 
in nature. The purpose of the study was to provide "baseline*^ data for 
determining the feasibility of further investigation into the use of 
quantitative judgmental data in evaluating school programs for determining 
program budget allocations. 

The Cost- Utility Analysis involved essentially two interrelated 
processes; program evaluation and program budget allocations. The primary 
function of the evaluation process was to derive a single valued numerical 
measure of the asymptotic level of program worth (utility) based on the 
judgment of preference or worth by several evaluators. Each e valuator 
rated on a continuous scale from 0 to 100 every program with respect to 
each of the given criteria • After each evalua^-or had assigned a value 
which represented the intensity of program worth to all program-criteria 
alternatives, his ratings were checked for internal consistency using 
the Churchman- Ackoff approximate measure of value. If the judgments 
and ratings were inconsistent, then the evaluator adjusted his ratings 
to conform to his judgment. 

The resource allocation . process utilized the composite program utility 
values derived from the evaluation process, the established program threshold 
costs and the program constrained maximums to generate optimal program 
expenditure levels which maximized the utility or worth of all programs. 

Conclusion 

The process of evaluating programs for purposes of resource allocation 
provides a transparent process for involving all persons of the school 
community in coming to some concensus about the relative worth or value 
of these educational programs. 

The Cost-Utility Analysis procedure stimulated the collection of 
program cost data not previously available to the budget administrator 
such as program threshold cost 5 which previously was unavailable although 
^'^ the project school district has had an operable PPBS accounting system 
for several years. 

The discrepancies between the recommended and actual program budget 
*^ allocations were small enough to justify a more comprehensive application 
of the program evaluation and budget allocation process. 

The procedures and model utilized in this investigation can be 
categorized as a judgment-based decision system. Fundamental to the 
decision system, but not directly demonstrable, are two implied axioms: 



ERIC 



33 



1. Judgments in a decision system should be fragmented into 

its least complex elements. This permits judgments to occur 

at the single element level with the opportunity to allocate 
judgment responsibility among decision makers, 

2. Decision makers can make explicit preference and value 
judgments. This being contingent upon appropriate scaling 
techniques and proper training. 

The question of validating judgment bassd decision systems must 
inevitably be approached. Validation in the classical sense consists of 
demonstrating the coherence between the output of a system, in this case 
program budget allocation determined through a cost-utility analysis, 
and the comparable output of some other system considered to be valid. 
Here the valid system may be assumed to be the budget administrator. This 
would appear to be the case in considering a measure of the judgmental 
gap between the output of the two comparable systems. This would be a 
valid approach provided the goal of the system was to reproduce the 
budget administrator's program allocation decisions. To some extent 
this has been an important consideration in testing th feasibility of 
the cost- utility analysis study. The importance of the cost-utility 
analysis is not simply that the results of the two comparable systems 
are essentially the same, but that the process caused more value to 
accrue and bear on the final outcome than if the outcome were solely 
or primarily determined by the budget administrator. 

In another sense the system must of its own right be self-validating 
provided the preference or value judgments can be considered *'true values.** 
It is this problem of systems validation in the non-classical sense which 
needs to be more thoroughly studied and researched. 

In conclusion, the Cost-Utility Analysis Process is a transparent 
process by which programs are evaluated and resources distributed to 
programs. It can provide school districts with a potentially powerful 
communication vehicle. Human judgment has prevailed in assessing program 
worth and in the determination of basic costs • Judgment is what both 
education and science a^'e all about. Th^ purpose of both these endeavors, 
as has been the purpose of this stud^*, is th^ refinement and improvement 
of human judgment. Judgment lies at the very heart of effective professional 
decision making. Rejection of the use of judgment for assessing educational 
outcomes will result in failure to consider the most important aspects 
of a school's educational goals and objectives. 
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APPENDIX A 



PROCEDURES 



Appendix A is divided in two parts. The first part will consider 
the theoretical development of the evaluation-resource allocation model, 
which will serve to allocate limited resources among competing educational 
programs based on the maximization of total program utility given a budget 
constraint. More formally this may be stated as: 

L 

Max Z U^(x^), U^(x^) = Utility of program i at expenditiu e 
i = l level Xj^, i = 1, 2, . * . , L. 

Subject to: N, Budget constraint. 

The second part will consider the design of a computerized man-machine 
system, which can be utilized to facilitate the application of the 
evaluation-resource allocation model. 

Evaluation Model 

In order to allocate limited resources among competing programs 
based on the utility of these programs, it is necessary first to 
establish a functional relationship between the utility of a program 
and the cost of each program at various levels of cost. The utility 
of a program is defined as a real valued number which expresses an 
evaluator's judgement of the degree to which a given program has the 
potential to contribute to the satisfaction of some stated criterion 
or objective. It is assumed that the evaluator's judgement or prefer- ^ 
ence takes into consideration both qualitative and quantitative factors. 
The amount of money or resources allocated to a given program will 
depend on the degree to which that program satisfies a given set of 
criteria or objectives. It is observed from the above discussion that 
utility is a function of level of expenditure and some set of criteria 
or objectives. 

A number of I'elationships between utility and level of expendi- 
ture become immediately obvious, which will enable us to determine 
a relatively good approximation of the cost-utility curve, and 
subsequently a functional relationship between program expenditure 
and program utility. These relationships or underlying assumptions are: 

1. A program allocated zero resources has zero utility for 
any set of criteria or objectives. 



Frank W, Banghart , Educational Systems Analysis (Toronto, 
Ontario: The Macmillan Company, 1969), p. 208* 
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The utility of a program increases as it's level of expenditure 
increases • 



3, There is a minimum level of program expenditure (threshold 
level) which must be allocated to a given program if it is 
to operate at all. Corresponding to this threshold level of 
program expenditure is a minimum level of acceptable utility. 
A utility value below this level implies a non-operable level 
of program expenditure, 

4. The utility of a program does not increase indefinitely as 

it is allocated greater and greater expenditure levels . This 
seems to be a reasonable assumption. For example, it is very 
doubtful that if teachers salaries were doubled, the value or 
worth (utility) of the program would also double. That is, 
beyond the threshold level of utility the marginal utility 
decreases as the expenditure level increases. In an economic 
context this assumption means that small allocations lead to 
essentially zero returns, while large ones have a saturation 
effect, the "law of diminishing returns." 

It follows from the above assumptions that the cost-utility curve 
passes through the origin of two dimensional space; lies entirely in 
the first quadrant; and is asymtotic to some maximum level of utility. 
The S-shaped curve in figure 7 satisfies all of the above assumptions. 
By an S-shaped function, we shall mean a non-decreasing function U(x), 
which is convex in (0,C£^), concave in (C£,<») and which is continuous 
at X = 0 and at x = C.. It is also assumed that U(C) = 0. 



A • 
-.1 
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Fig. 7. The S-shaped Curve 
Assumption four states that beyond the threshold level of utility, 
U(x£), the marginal utility decreases as program expenditure increases. 
Since the S-shaped curve has an inflection point, A, abo/e which the 
marginal utility decreases as expenditure increases, let us assign the 
coordinates of point A(Cj^, U(x.)), where is the threshold level of 
program expenditure and ^(x^) is the threshold level of program utility. 

The rationale for assuming that the threshold level of program 
expenditure falls on the inflection point, A, of the program cost- 
utility curve follows from economic theory (see figure 7.1). 

Figure 7.1(a) represents the typical program cost-utility curve 
with the inflection point at A. Figure 7.1(b) shows both the marginal 
utility (MU) and average cost (AC) curve. The MU curve will intersect 
the AC curve at its maximum value, E. Point C is the point on the 
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cost- utility,' curva in v/'hich the marginal utility p^r unit cosr is so 
small as to vrarrant no further finding. Points B ar.d C in •■"ir^ure 
7.1(a) divide the cost-utility curve into three sections, I, II and III. 
It is observed that the rational decision naker ;7ill not operate in 
either section I or section III, but in section II. In section I, AC 
is increasing to its maxirr.um. That is, the AC per unit utility is :it 
its highest. In section III the AC is decreasing slowly and then increa- 
sing, while the MU per unit cost is approaching zero. The per unit cOot 
produces negligible added utility. The best trade off betv;een average 
cost and total utility lies in section 11."^ 

Now since point A lies just outside and approachas the rational 
allocation area 11, it seems a reasonable assumption that the threshold 
cost, C^, be the abscissa of point A. 




Fi^. 7.1. The ?-j !.-if:lo:":Ghip ii-^^ '.v.-een StuC::jr.t 
L:«;j':rndi "ur:? for Pro^r.:-:- i 



Richard H. Leftwich, The Price Systen and RetSoi:rcG Allocaticn 
(Mew York; Holt, Rinehart and Winston^ 1966), pp. 113-115. 
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The logarithmic (reciprocal transformation) function gives a 
very good approximation to the underlying assumptions and S-shaped 
curve depicted in figure 7. The functional relationship between 
utility and cost then is as follows (see figure 8). 



0 

3i/2 = C. 

Fig. 8. A Functional Relationship Between Cost and Utility 

U^Cx^) = u.e = Utility of program i at expenditure level 

^ X^. i = 1, 2, - . m. = x^ = N. 

6; = 2C.5 where C. is the threshold level of expenditure 
for program i. 

= asymptotic level of program v/orth or value for program i, 

ACC^jUCx^)) = the inflection point on the cost-utility curve. 

Aggregating additive utilities 

As stated earlier the utility of a program is a function of some 
set of criteria or objectives. Most allocation decisions will be 
based on such multi-criteria situations. The problem is one of evaluation. 
That is, how to obtain single valued comparisons of the utilities of 
program alternatives when each utility must take into account the 
contributions of several criteria or objectives. Additive utility 
theories offer one possible approach to this problem. For example, 
suppose program utility is a function of the following four criteria; 
Relevance, Generality, Assimilation and Urgency. Relevance is defined 
as the degree to which a program has the potential to satisfy the social 
and cultural goals and objectives of the institution. Generality and 
Assimilation are economic factors which assess the interrelatedness of 
all programs. Generality is the degree to which the outputs of a given 
program are utilized by the other programs, while Assimilation is the 
degree to which a program utilizes the outputs of the other programs. 
Urgency is a political criterion which gives explicit recognition to 
pressure groups in the budgeting process. Now consider the evaluation 
matrix in figure 9. 
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sEvaluator n 



s — 































Evaluator 2 
Evaluator 2 



Program 1 
Program 2 



• * • 



Program L 



Relevance Assimilation 

Generality Urgency 

Fig. 9. The Evaluation Matrix 



= Asymptotic level of program worth or value , 
i = Program 

j = Criterion of evaluation 
k = Evaluator 
L = Number of programs 
m = Number of criteria 
n = Number of e valuators 

= Weight given to criterion j 



m 

Z w. = 1. 

j = l ^ 

For any two programs in a. set of alternative programs, if the 
first program is evaluated to be of greater worth with respect to a 
given cr.rteria than the second program, then the utility of the first 
program is greater than the utility of the second program. This follows 
from Theorem 1: ^ 

A number, U(x)5 can be assigned to each x in X 
so that if X and y are in X, then 

< 

X y iff U(x) = U(y), 

Theorem 1 permits one to go back and forth between judgements of worth 
or value and utilities, where utility was defined previously to be 



1 

• Fishburn, "Utility Theory," p. 3^^. 
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a real valued number. This implies that the entries, -j^j "^^^ 
evaluation matrix are real valued numbers whose magnitu^^ determines 
the relative worth of the alternative programs with respect to a given 
criterion. That is, 

^iik " Utility assigned to criterion j in 
program i by evaluator k. 

^ If we assume the interpersonal conparability of utility for 
n = 2, then 

n 

T. U. /n = Utility assigned to criterion j for program 
k=l i by n evaluators. 



By Theorem 2A, which states: 



< n 



iff Z w.v. (x.) = Z w.v (y.) 
i=l ^ ' ' i=l ^ ^ " 



X y 
it follows that: 
n 

1. Z (U^.j^/n)w. = Weighted utility assigned to criterion 
k=l j for program i. 

m n 

2, ^ Z (13.. /n)w. = U. = Asymptotic level of value or 
j=l k=l '^^'^ ^ worth for program i. 

A methodology has thus been established by which to obtain single 
valued comparisons of the utility of program alternatives when each 
utility is an aiTalgamation of multi-criteria. In addition a rank order- 
ing of program alternatives has been established (Ordinal Utility). 
This procedure would suffice if we were interested only in selecting 
that program judged to have the greatest asymptotic level of program 
utility. This is however, not the case when we are interested in dec- 
isions by which to allocate scarce resources among competing educational 
programs. The resource allocation problem requires at least an interval 
scaling of program utilities with respect to the given set of criteria 
or objectives (Cardinal Utility). 

An interval scale can be obtained by first assigning an arbitrary 
point of origin, which shall be designated "0**, and an arbitrary unit 
of measure* The range of the interval scale will arbitrarily have 100 
units. 

Estimating^ Additive Utilities 

Program evaluation involves the assigning of utilities to program 
alternatives. These quantitative measures represent the relative worth 
of program alternatives with respect to a given criterion or objective. 



1 

Fishburn, "Utility Theory,'' pp* 346-347, 
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What is required is an efficient and logically consistent methodology 
for estimating these assigned utilities. Fishburn has classified under 
four aspects the bases for selecting methodologies for estimating 
additive utilities.^ The methodologies for estimating utilities with 
multi-dimensional criteria in the present evaluation model will consider 
the following aspects; 

1. Non-probabilistic estimation methods. 

2. Estimation methods which consider the preference- 
indifference relation, 

3. Estimation methods which evaluate factors (criteria) 
singly and independent of each other. 

4. Estimation methods which consider the levels of each 
criterion or objective to be continuous. 

Prior to assigning individual utilities to program alternatives, 
decisions must be made which identify and specify programs and criterion 
to be included in the evaluation matrix as well as the weighting of the 
criteria and the identification of the evaluators. It is also necessary 
to determine the threshold levels of program expenditure for each 
program identified in the evaluation matrix and the budget constraint, N. 

The development of this evaluation-resource allocation model will 
not be concerned with either the specific identification of programs 
and criteria or the selection of evaluators to be included in the 
evaluation model. It is suggested that these decisions will be peculiar 
to a given institution and will be taken as inputs into the evaluation 
model. Similarly, the threshold levels of program expenditure, C^, will 
be assumed as input for the evaluation-resource allocation model. 

Once the programs and evaluators have been identified and the thresh- 
old levels of program expenditure have been determined, decisions must 
be made as to which criteria will be included in the evaluation phase. 
If a single evaluator is involved, he simply identifies the 'criteria for 
evaluation and the Relative importance (weight) assigned to each criteria. 
If the selection of criteria and the assigning of weights to criteria 
are a group decision, the Delphi methodology may be employed to obtain 
group concensus.^ Assuming the criteria and weights have been satis- 
factorily establishled, each decision maker is now required to establish 
the relative utility of all programs for each criteria. Each individual 
decision maker proceeds in the following manner: 

1. The first procedural step is to have each individual 
decision maker rank order the program alternatives 
against each criteria. 



Fishburn, ''Methods of Estimating Additive Utilities,'' pp. ^36-^+39. 
Till, op.cit., pp. 57-60. 
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2, Next each individual is asked to assign a nuinber 
(utility) to each program alternative for each 
criteria between 0 and 100, such that these numbers 
are consistent with step one. Step 1 and step 2 
may be combined by eliminating step 1 and directly 
assigning a utility to each program alternative and 
criteria. It is however, probably easier to assign 
utilities once the programs are ordered with respect 
to a given criterion. 

3, The third and final step is to check the logical 
consistency of judgements of worth and value and 
the numerical representations of these judgements. 
If there are inconsistencies, then the appropriate 
utilities are changed to accommodate the appropriate 
judgement. The methodology judged most suitable 

for step 3 is the Churchinan-Ackof f approximate measure 
of value. 



For example, suppose the following utilities have been assigned to 
four programs considering only the criterion of Relevance • Assume also 
the threshold levels of program expenditure had been determined prior 
to assigning these utilities (see table 1). 



INITIAL PROGRAM-CRITERION MATRIX 



Criterion 



I Program 1 
Program 2 
Program 3 



Program H 



Relevance 

50 
10 
60 



40 



Table 1. 



First, the programs are ordered from most relevant to least relevant. 
The order may be determined from either step 1 or by ordering the numbers 
in the cells from largest to smallest . It is observed that the programs 
are ordered with respect to relevance as P3, P^ , P^, Pj. Now determine 
which is more relevant, or the combination of P., P^^, and P2. If 
P^ is more relevant, then the utility assigned to should be greater 
than the sum of the utilities assigned to P-j^, P^, and P2* is observed 

that we have an inconsistency since 60 is less than 50+10+40 = 100. 
Consequently, the utilities for P^, P,^ and P2 must be adjusted to conform 
to the given judgement while maintaining the relative magnitude of their 
utilities. An appropriate change might be to let P^^ = 25, P^^ = 20 and 
?2 = 10- Now the assigned utilities conform with the a priori judegment. 
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That is, 60 is greater than 25+20+10 = 55, 



If on the other hand the combination of P^j and is judged 
more relevant than P3, the numbers assigned as in table 1 are consistent 
since 60 is less than IOC. But the check for consistency is not complete. 
The following test for inconsistency must also be applied. 

1. Determine which is more relevant P3 or P^^ and P^, 
Suppose Pj^ and Pi| are preferred to It is seen 
that "the numbers assigned are consistent since 90 
is greater than 60. If P3 was judged more relevant 
then an adjustment in the assigned utilities must 
be made, 

2, Determine which is more relevant Pj^ or P^ and P2* 
Suppose and P^ + P2 = 20 + 10 equals 30 

is greater than 25, hence the utilities assigned are 
consistent. 

In sunmiary, suppose the following judgements had been made: 

1. P^ + Pi^. + ^2 ^o^^ relevant than P3. 
Check for consistency: 

Is 50 + 40 + 10 greater than 60? Yes, consistent. 

2. ^1+^1+ niore relevant than P3. 
Check for consistency: 

Is 50 + ^0 greater than 60? Yes, consistent. 

3. ^4 + relevant than P^. 
Check for consistency: 

Is 40 + 10 greater than 50? No. 
Change P2 = 20. 



FINAL PROGRAM-CRITERION MATRIX 



1 Criterion 




1 

Relevance ! 


j Program 1 


r 

! 

50 1 


Program 2 


20 \ 


1 Program 3 
1 


60 J 


i P u 

1 Program 4 





1 



Table 2. 

Er|c a< 



Resource Allocation Model 



The rf^sults of the previous sections have provided a functional 
relationship between the utility, U^(x-), and the level of expenditure, 
Xi, for an program i, i = 1, 2, . . . L. That is, given C^^, i = 1, 
2, • . . , L, and the correspo.iding values U^(xj^), i = 1, 2, . , ., L 
derived by methods outlined in the evaluation model, it is assumed that 
U^Cx-^) and are functionally related by the logrithmic function. 



U.(xi) = U.e 



'^i/x. 



The problem now is to find the values x^, 1 = 1,2, . . .L, constrained 
by N dollars allocated amongst L programs such that 

L 

[ the sum, E U^(xj^), is maximized, i.e., 
i=l 

X,) (1) 



L 




Ex. 


= N 


1 = 1 






L 


MAX: 


Z 




i=l 


S.T. : 


N 



(2) 



or more completely: 



L 

1 



MAX: Z U,e ^'''^i (3) 



i=l 

L 

Z 

i=l 



S.T. : Z = N (4) 



For example, consider the case where L = 1. By the method of 
Lagrange multipliers, a general method for finding an extreme value 
(maximum or minimum) of a function subject to one or more constraints. 

2 -B., 2 1 

F(x F(x.,x ,X), = Z U.e ^^^i+X( Z x^ - N). 

i=l ^ i=l 

2 2 -i/x. 

That is, the objective function, Z U.(x.) equal Z U.e ' 

i=l ^ i=l ^ 

2 

and the constraint function, Z x^ equal N, are written as a single 

i=l 

function, F, using the Lagrange multiplier, X. The values x^ and X2 
are then found by finding the global maximum on F(xj^, Xj, X). This is 



^Samuel B. Richmond, Operations Research for Management Decisions 
(New York: The Ronald Press Co. , 1968), pp. 112-122. 
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done by taking the partial derivative of F with respect to x^, Xj, and A; 
setting the partial derivatives equal to zero and solving for x^^ and X2, i.e., 

2 -i/x. 2 
F = Z U.e A( Z X. - N), for i = 1, 2. and 



i=l ^ i=l " 



< x^ < N. 
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Taking the partial derivative of F with respect to Xj^ , and X, we get 

"^i/x. 

aF = u-e - 1 +X = 0. 

^ -3i/x. 2} + X = 0 

2 -3./ 
= 3i/x^ U.e ^^""i + X = 0. 

For i = 1, 

3F 2 -3.. 

Sxj^ = B^/x^ U^e ^^'^l + X = 0. (5) 

For i = 2, 

3F = gg/x^ Ue ''''^2 + X = 0, (6) 
8X2 

For X, 

9F + X, + x^ = N (7) 

Equations (5), (6), and (7) are respectively equivalent to equations 
(8), (9) and (10)> 

6 /x2 U.e ^^/^ = - X. C8) 

111 

X + = N (10) 
1 

Solving the three equations, (8), (9)> and (10), simultaneously, we get 



All 



r 2 



but this is simply equivalent to 



-0 



1/x, 



(11) 



(12) 



That is, E U-(x-) is maxiinized when the first derivative of with 

i=i 

respect to is equal to the first derivative of with respect to 
X2 and x^ + equal N. This can be interpreted geometrically as follows 
(see figures 10a and 10b). Consider the cost-utility curves for two 
programs, and Pj, figure 10a„ In figure lOb, line L, is moved up 
or down until a + B = N, th 



dui 



dx. 



du. 



dXr 



e budget constraint. It is obse:?ved that 
at points A and B. 



Consequently, a dollars shoi\ld be allocated to program and b dollars 
should be allocated to program P^. 



U, 




Figura 10a. Program cost-utility curves 
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Figure 10b. First Derivative curves of co3t-util.i ty functions 

Although the optimal solution was derived for only two programs, 
the theory extends to L programs • In this case the optimal solution 
will occur when the following two conditions are simultaneously met: 

(13) 



The technique for optimising total program utility with the 
given cost constraints as previously developed assumes that the du^ 

exist such that for i = Ij 2, . • L, conditions (13) and 

(1^) can be satisfied. It is observed, however, that sit- dx^ 
uations will occur in which it is impossible to satisfy both 
(13) and (1^). When this incongruity does occur > the procedure is to 
fund that program i which has the minimum maximum first derivative at 
its threshold level of program expenditure; subtract from N and 
maximize total program utility for the remaining L - 1 programs svibject 
to (N - Cj_) dollars. For example, suppose we initiate the following 
optimization problem: 

L 

MAX: I U.(x.) for i = 1, 2, . . . , L (15) 
i=l ^ 

L 

i.T.: I X. = N. (16) 

1:^1 ^ 



du^ du^ 



dx. 



dx. 



du 



dx 



for 



x^ + X2 + * 



+ x^ = N 
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If the conditions imposed on the problem are such that (15) and 
(16) cannot simultaneously l^e satisfied, then fund that program i whose 
cost-utility curve has the smallest maximum marginal utility at a level 
of dollars and proceed as follows: 

L-1 

MAX: E U.(x.), for i = 1, 2, . . . , L-1 (17) 
i-1 ^ ^ 



L-1 

i-1 



S.T*: E = (N - C^) (18) 



This procedure is iterated until both of the conditions for optimization 
can be simultaneously satisfied. . 

This concludes the theoretical considerations for the evaluation- 
resource allocation model with the exception that there is not an 
analytic solution to equations (13) and (14), which will provide us with 
the- optimal allocations X£,i = l,2, ...,L. This, however, is 
not a serious restriction as the values x^, may be obtained by 
recursive ipethods. 

System Design 

The entire process of program evaluation and resource-allocation 
is computerized to allow the decision Tnaker(s) to interact via machine 
to quickly determine the values of the decision variables. The following 
subroutines have been programmed to meet the above objective. 



Program 



The Program subroutine enables the user to enter by name the programs 
to be evaluated. The maximum number of programs that can be named is 12. 

Criteria 

This subroutine permits the user to enter by name the criteria 
against which the program alternatives will be evaluated. The maximum 
niimber of criteria that can be named is 12. 

Evaluate 

The Evaluate subroutine allows each evaluator to enter his assigned 
utility rating for each criterion for as many programs as will require 
budgeting. Each of the evaluator 's ratings are checked for logical 
consistency against his preferred judgement using the Churchman- Ackoff 
method of approximating measures of value. The maximum number of eval- 
uators per routine is 20. 



Ralph H. Pennington, Introductory Computer Methods and Numerical 
Analysis (Toronto,. Ontario: The Macmillan Co., 1970), p, 286. 
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Weight 

The Weight subroutine permits the user to weight each criterion 
such that the sum of the weights of all criteria is equal to 1. If 
the user does not wish to weight criteria, it is assumed that all criteria 
have equal weight. 

Threshold 

This subroutine requires that the user enter a value, Cj^, for each 
program. is the threshold ecpenditure for program i. 

Plot 

The Plot subroutine allows the user to print out on terminal a 
plot of the cost-utility curves for as many programs as desired. The 
cost-utility curves may be plotted individually or collectively. 

Constraint 

This subroutine requires the user to specify the budget constraint, 
N, which will be utilized to maximize total program utility* 

Matrix 

The Matrix subroutine gives a printout of the optimal expenditures 
to be budgeted to each program and a corresponding value of program 
utility. The subroutine also gives a value which represents total 
program utility. This value is found by summing over the individual 
utilities which correspond to the budgeted level of expenditure. 

Maximum 

This subroutine allows the user to place a maximum value on the 
amount of dollars allocated to any of the named programs. 

Restart 

The Restart subroutine clears all existing storage units and permits 
the user to start anew the evaluation-resource allocation process. 

Stop 

This subroutine allows the user to exit from the evaluation- 
resource allocation program. 

The evaluation- resource allocation on-line system is programmed to 
allow maximum flexibility for determining the values of the decision 
output variables. The initial run through the system is linear with 
the exception of the two printout options. Matrix and Plot. After the 
initial run, the user has the option of choosing any one of the twelve 
subroutines contained in the program in order to change any of the 
systems variables (see Fj-ow diagram of evaluation-resource allocation 
system on the following page). 
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Having derived the optimal budget allocations which maximize cotal 
program utility, post optimal sensitivity analysis may be performed by 
calling any of the input subroutines (Weight, Threshold or Constraint); 
changing the value(s) of the input variables and observing the new 
optimal program budget allocations. 
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INTRODUCTORY 
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Tig. 11. Flow for Program Evaluation and Budget Allocation 
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PROGRAM 








WEIGHT 



CRITERIA 



1 



EVALUAT 




Fig. 11. Conrinued 
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CRITERIA 




r 




yes 



THRESP 


lOLD 




r 


WEIGHT 


I 


EVALl] 


AT 



PROGRAM 



Fig. 1.1. Continued 
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Fig. 11. Continued 



APPENDIX B 



ACTIVITIES AND TIME SCHEDULE 



Activity 



Starting Time 



Completion Time 



September 1, 1972 
September 16, 1972 
October 1, 1972 



1. Meet and discuss operational 
plans with principal and 
assistant principal of 
Pennsbury High School 

2. Make arrangements for 
installation of a computer 
terminal at Pennsbury 
High School 

3. Determine programs 

4. Determine criteria 

5. Determine individual 
program threshold costs 

6. Make arrangements to tie 
in with the Lehigh University 
Computer Network 

7. Assign relative weights 
to criteria 

8. Determine constrained 
budget 

9. Have decision-makers assign 
numerical values to program 
criteria matrix 

10 • Submit data to interactive 
computer terminal to obtain 
recommended program expenditure 
allocations for ensuing year January 6, 1973 

11. Have budget administrator 
determine actual program 
budget allocation for 

ensuing year February 26, 1973 

12. Determine degree of 

judgmental gap March 16, 1973 



September 1, 1972 September 15, 1972 



September 30, 1972 
September 30, 1972 
October 15, 1972 



October 1, 1972 October 31, 1972 



October 1, 1972 
October 16, 1972 
November 1, 1972 



October 31, 1972 
October 31, 1972 
November 15, 1972 



November 16, 1972 November 30, 1972 



February 25, 1973 



March 15, 1973 
March 31, 1973 
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APPENDIX C 
ASSIGNING CRITERION WEIGHTS 



We. will assume for the purpose of this exercise that the following criteria: 
relevance, generality j assimilation , urgency and need, are to be evaluated 
against the major objectives of the school district- 

DEFINITION: 



Relevance 



Generality 



Assimilation 



Urgency 



Need 



Utility 



The degree to which a program satisfies the social 
and cultural goals and objectives of the institution. 

The degree to which the outputs of a given program 
are utilized by the other programs* 

The degree to which a program utilizes the outputs 
of all other programs. 

The degree to which an evaluator wishes to give 
explicit recongition to pressure groups in the 
budgeting process with respect to a given program. 

The degree to which en evaluator wishes to BiTiphasize 
a higher priorit}'- because of special circumstances. 

A real valued number that .expresses an evaluator 's 
judgment of the degree to which a given program has 
the potential to contribute to the satisfaction of 
some stated criterion or objective. 



CRITERIA EVALUATION EXERCISES: 

1. Rank order the criteria. 

2. Assign a number between 0 and 100 to each criterion, so that these 
numbers are consistent with your rank order. The sum total not to 
exceed 100. 

3. Check for inconsistencies between your judgments and the numerical 
representatives of your judgment. 
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RANK 


CRITERIA 


UTILITY 


1 






2 

3 










5 





EXAMPLE; 

1. Rank order Criteria Need 40 

Generality 30 

Relevance 15 

Assimilation 10 

Urgency 5 

2. Assign values consistent with rank order. 



3. Check for inconsistency. 



APPENDIX D 



All Instructional Categories'' 
(Supplies, Textbooks, Audio Visual Aids) 



Program 


71-72 




72-73 


Average 


English 


4858.27 




5412.36 


5135.31 


Social Studies 


5857.57 




9635.60 


7746.58 


Mathemat ics 


1023.11 




2006.46 


1514.78 


Science 


8681.13 




10,597.76 


9639.44 


Business Education 


6104.20 




4615.04 


5359.68 


Language 


1422.54 




908.52 


1165.53 




Instructional Sub-Category 






(Textbooks) 






Prograni 


•'^71-72 


Enrollment ?«'f73-74 


Enrollment 


English 


3535.10 


2438 


2000 


2384 


Social Studies 


5559.00 


2180 


2000 


2406 


Mathematics 


2006.43 


W13 


2000 


1327 


Science 


4843.40 


1275 


2500 


1420 


Business Education 


3537.75 


1335 


2000 


1335 


Language 


908.52 


804 


1200 


658 


Dollar figure 


represents 


threshold 


cost and program growth 


Dollar figure 


represents 


threshold 


cost. 
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APPENDIX E 



ASSIGNING PROGRAM - CRITERION VALUES 



We will assume for the purpose of this exercise, that six programs; 
Business Education, English, Language, Mathematics, Science and Social 
Studies are to be evaluated against the five criteria; Relevance, Generality, 
Assimilation, Urgency and Need. 



DEFINITION: 



Relevance 
Generality 
Assimilation 
Urgency 

Need 
Utility 



Threshold Cost 



The degree to which a program satisfies the social 
and cultural goals and objectives of the institution. 

The degree to which the outputs of a given program are 
utilized by the other programs. 

The degree to which a program utilizes the outputs of 
all other programs. 

The degree to which an evaluator wishes to give explicit 
recognition to pressure groups in the budgeting process 
with respect to a given program. 

The degree to which an evaluator wishes to emphasize 
a higher priority because of special circumstances. 

A real valued number that expresses an evaluator' s 
judgment of the degree to which a given program has 
the potential to contribute to the satisfaction of 
some -stated criterion or objective. 

The minimum level of program expenditure which must be 
allocated to a given program if it is to operate at all. 



Constrained Maximum The dollar- value, above which, a specified program can- 
Maximum not effectively utilize its resources. 



Program Evaluation Exercise 

Assign values to each cell of the Program- Criterion Matrices on the last 
page according to the following procedure : 

(1) Rank order the program alternatives for the first criterion. 

(2) Assign a number between 0 and 100 to each program alternative, 
such that these numbers are consistent with (1). 

(3) Check for inconsistency between your judgments and the numerical 
representation of your judgments. 



EXAMPLE: 



Rank order 
programs 
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Relevance 
English 90 
Social Studies . 75 
Science 70 
Mathematics 60 



El 



2. Assign values consistent 
with rank order 

3. Check for inconsistency 



PROGRAM - CRlTl-RlON MATRIX 



RANK 


PROGRAM 


UTILITY 


1 






2 






3 






4 






5 






6 







GENERALITY 



RANK 


PROGRAM 


UTILITY 


I 






2 






3 






/♦ 






5 






fi 








ASS IM TUT ION 


RANK 


PROGRAM 


UTILITY 


1 






2 






3 






4 






5 






6 







URGENCY 




RANK 


PROGRAM 


UTILITY 


L 






2 






3 






4 






5 






6 






NEED 


RANK 


PROGRAM 


UTILITY 


1 






2 






3 






4 






5 






6 
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CUHRICULUM ClUIRMAN (ENG.) 

pkoc;ram - cRiir.RioN matrix 



APPEIIDIX F 



RELEVANCE 



RANK 


PROGRAM 


UTILITY 


1 


ENGLISH 


95 


2 


SOCIAL STUDIES 


80 


3 


SCIENCE 


65 


4 - 


MATH 


60 




LANGUAGE 


55 




BUSINESS EDUCATION 


55 


GENERALITY 


HANK 


PROGRAM 


UTILITY 


1 


ENGLISH 


90 


2 


SOCIAL STUDIES 


75 


3 


SCIENCE 


60 




^tATH 


55 


5 


LANGUAGE 


50 


6 


BUSINESS EDUCATION 


45 




ASSIMIUTION 


HANK 


PROGRAM 


UTILITY 


1 


ENGLISH 


90 


2 


SOCIAL STUDIES 


85 


3 


SCIENCE 


80 


4 


MATH 


70 


5 


RTTf?T?»resS EDTTHATTON 


6S 


6 


LANGUAGE 


60 


URGENCY 




RANK 


PROGRAM 


UTILITY 


I 


ENGLISH 


80 


2 


SOCIAL STUDIES 


75 


3 


SCIENCE 


70 


4 




65 


5 


L.\NGUAGE 


60 


6 


BUSINESS EDUCATION 


55 




NEED 




RAtTK 


PROGRAM 


UTILITY 


1 


ENGLISH 


80 


2 


SOCIAL STUDIES 


75 


3 


SCIENCE 


70 


4 


^L^TH 


65 


5 


LANGUAGE 


60 


6 


BUSINESS EDUCATION 


55 



CURRI CULUM CHAI R MAN (S . S . ) 
PROGRAM - C:RITi!R10N HA T RU . 



RELEVANCE 



RANK 


PROGRAM 


UTILITY 


1 

<L 






2 


ENGLISH 




3 




ao 


4 


STTKVCE 


75 


e 


BUSINESS EDUCATION 


60 




LANGUAGE 


55 




GENERALITY 


RANK 


^ PROGRAM 


UTILITY 


1 


ENGLISH 


100 


2 


SOCIAL STUDIES 


90 


3 


SCIENCE 


85 


4 


LANGUAGE 


70 


5 


MATH 


65 


6 


BUSINESS EDUCATION 


50 




ASSIMILATION 


RANK 


PROGRAM 


UTILITY 


1 


BUSINESS 


100 


0 

& 


SOCIAL STUDIES 


95 


3 


ENGLISH 


90 


4 


LANGUAGE 


70 


5 


IMTH 


65 


6 


SCIENCE 


60 


URGENCY 


RANK 


PROGRAM 


UTILITY 


1 


SCIENCE 


100 


2 


BUSINESS EDUCATION 


95 


3 


SOCIAL STUDIES 


80 


4 


ENGLISH 


75 


5 


LANGUAGE 


60 


6 


MATH 


55 




NEED 


RANK 


PROGRAM 


UTILITY 


1 


SOCIAL STUDIES 


100 


2 


BUSINESS EDUCATION 


96 


3 


SCIENCE 


80 


4 


MATH 


70 


? 


ENGLISH 


60 


6 


LANGUAGE 


50 
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CURRICULUM CHAIRMAN (M) 
PUOGRAM - CRITERION MATRIX 



RELEVANCE 



RANK 


PROGRAM 


UTILITY 


1 


ENGLISH 


100 


2 


SOCIAL STUDIES 


90 


3 


BUSINESS EDUCATION 


80 


4 


SCIENCE 


75 




MATH 


70 


n- 


LANGUAGE 


50 


GENERALITY 


RANK 


PROGRAM 


UTILITY 


I 


ENGLISH 


100 


2 


MATH 


95 


3 


SCIENCE 


90 




BUSINESS EDUCATION 


80 


5 


SOCIAL STUDIES 


70 


6 


LANGUAGE 


60 




ASSIMIUTION 


RANK 


PROGRAM 


UTILITY 


1 


SCIENCE 


100 


2 


SOCIAL STUDIES 


85 


3 


. MATH 


75 


4 


ENGLISH 


70 


5 


LANGUAGE 


65 *• 


6 


BUSINESS EDUCATION 


60 


URGENCY 




RANK 


PROGRAM 


UTILITY 


I 


SOCIAL STUDIES 


100 


2 


ENGLISH 


90 


3 


SCIENCE 


80 


4 


T/\KGUAGE 


75 


5 


BUSINESS EDUCATION 


70 


6 


MATH 


65 




NEED 




RAMX 


PROGRAM 


UTILITY 


I 


ENGLISH 


100 


2 


^LATH 


90 


3 


SCIENCE 


85 


4 


LANGUAGE 


75 


5 


BUSINESS EDTjCATION 


70 


6 


SOCIAL STUDIES 


65 



o 
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CURRICULUM CIL\IR?,UN (SCI.) 
VUOGilAM - ClUTL-RiON MATRIX 



RELEVANCE 



RANK 


PROGRAM 


UTILITY 


1 


ENGLISH 


100 


2 


SOTIAL STirDTF.S 


95 


3 


SPTKNTE 


85 


4 


NUTH 


80 




rtt<;tvp:«?r FniirATTON 






LANGUAGE 


30 


GENERALITY 


RANK 


PROGRAM 


UTILITY 


I 


ENGLISH 


i '^0 


2 


^UTH 


90 


3 


SCIENCE 


85 


4 


SOCIAL STUDIES - 


60 


5 


LANGUAGE 


20 


6 


BUSINESS EDUCATION 


10 




ASSIMILATION 


RANK 


PROGRAM 


UTILITY 


1 


SCIENCE 


100 


2 


SOCIAL STUDIES 


80 


3 


ENGLISH 


50 


4 


BUSINESS EDUCATION 


40 


5 


L^\NGUAGE 


30 


6 


MATH 


20 


URGENCY 


RANK 


PROGRAM 


UTILITY 


I 


SCIENCE 


100 


2 


SOCIAL STLT)IES 


80 


3 


ENGLISH 


70 


4 


BUSINESS EDUCATION 


60 


5 


L,\NGUAGE 


50 


6 


MATH 


40 




NEED 




RANK 


PROGRAM 


UTILITY 


1 


SCIENCE 


100 


2 


BUSINESS EDUCi'.TION 


SO 


3 


SOCIAL STUDIES 


70 


4 


ENGLISH 


60 


5 


mm 


50 


6 


L/\NCUAGK 


40 
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COORDINATOR (B,E. ) 
PROGRAM - CRITKUION MATRIX 



REU VANCE 



RANK 


PROGRAM 


UTILITY 


1 


BUSINESS EDUCATION 


100 


2 


ENGLISH 


90 


3 


SOCIAL STUDIES 


85 


4 


mm 


80 




SCIENCE 


50 


hi- 


LANGUAGE 


25 


GENERALITY 


RANK 


PROGRAM 


UTILITY 


1 


ENGLISH 


100 


2 


MATH 


95 


3 


SOCIAL STUDIES 


80 


4 


LANGUAGE 


25 


5 


SCIENCE 


20 


6 


BUSINESS EDUCATION 


15 




ASSIMILATION 


RANK 


PROGRAM 


UTILITY 


1 


BUSINESS EDUCATION 


100 


2 


SCIENCE 


95 


3 


MATH 


85 


4 


SOCIAL STUDIES 


80 


5 


ENGLISH 


50 


6 


LANGUAGE 


25 


URGENCY 


RANK 


PROGRAM 


UTILITY 


1 


ENGLISH 


100 


2 


MATH 


95 


3 


BUSINESS 


90 


4 


SOCIAL STUDIES 


50 


5 


SCIENCE 


25 


6 


LaNGUAGE 


10 




NEED 




RANK 


PROGRAM 


UTILITY 


1 


BUSI.NtSS EDUCATION 


100 


2 


ENGLISH 


95 


3 


SOCIAL STUDIES 


90 


4 


MATH 


80 


5 


SCIENCE 


50 


6 


LANGUAGE 


25 



COORD INATOR (UNC. ) 
PK(X;KA.^ ■ LiU ' il.KlU N MATRIX 



REUV/NCE 



^RANK 


PROGEIAM 


UTILITY 


L 


ENGLISH 


100 


.2 


SOCIAL STUDIES 


95 


3 


BUSINESS EDUCATION 


90 


4 


LANGUAGE 


75 




SCIENCE 


30 




MATH 


10 


GEl^RALITY 


RANK 


PROGRAM 


UTILITY 


_ I 


EJIGLISH 


100 


2 


SOCIAL STUDIES 


50 


3 


BUSINESS EDUCATION 


40 


U 


LANGUAGE 


30 


^ 


MATH 


20 


6 


SCIENCE 


10 




ASSIMILATION 


RANK 


PROGRAM 


UTILITY 


l- 


ENGLISH 


100 


2 


BUSINESS EDUCATION 


50 


3 


LANGUAGE 


40 




SOCIAL STUDIES 


30 


_ 5 




20 • 


6 




10 


URGENCY 


JIANK 


PROGRAM 


UTILITY 


- 1 


ENGLISH 


100 


2 


BUSINESS EDUCATION 


50 


3 


SOCIAL STUDIES 


40 


4 


LAXCUAGE 


30 


5 


SCIENCE 


20 


6 


MATH 


10 




MEED 




RANK 


PROGRAM 


UTILITY 


1 


ENGLISH 


100 


2 


BUSINESS EDUCATION 


50 


_ 3 


<;orTAL STIFDTE?; 


40 


1 4 


T.AKnrAGE 


30 


5 


SCIENCE 


20 


6 


MATH 


10 



F6 



ERIC 







DTnV*T"J* 

dULHj£T 




PROGRAM - CRITERION MATRIX 










RANK 


PROGRAM 


UTILITY 


1 


ENGLISH 


100 


2 


SOCIAL STUDIES 


95 


3 


lilATH 


85 


4 


SCIENCE 


80 




BUSINESS EDUCATION 


70 




LANGUAGE 


60 


GENERALITY 


RANK 


PROGRAM 


UTILITY 


I 


ENGLISH 


100 


2 


SOCIAL STUDIES 


95 


3 


MATH 


90 


4 


SCIENCE 


85 


5 


BUSINESS EDUCATION 


80 


6 


LANGUAGE 


. 70 




ASSIMILATION 


RANK 


PKOGRAM 


UTILITY 


1 


SCIENCE 


100 


2 


RUSINESS EDTIT/^TION 




3 


LANGUAGE 


80 


4 


AWTH 


70 


5 


SOCIAL STiroiKS 


60 


6 


ENGLISH 


50 


URGENCY 


RANK 


PROGRAM 


UTILITY 


I 


ENGLISH 


100 


2 


SCIENCE 


90 


3 


MATH 


85 


4 


BUSINESS EDUCATION 


80 


5 


SOCIAL STUDIES 


75 


6 


I.^XGUAGE 


60 




NEED 




RANK 


PROGRAM 


UTILITY 


1 


ENGLISH 


lOO 


2 


SCIENCE 


95 


3 


SOCIAL STUDIES 


75 


4 


BUSTOTSS EDUCATION 


70 


5 


MATH 


50 


6 


LANGUAGE 


40 



PROGRAM - CRITKRION MATRIX 
COMPOSITE EVALUATION MATRIX. 

RELEVANCE 





PROGRAM 


UTILITY 


1 


ENGLISH 


86.25 


2 


SOCIAL STUDIES 


92 .00 


3 


MATH 


68. 7S 


4 


SCIENCE 


67.50 


5 


BUSINESS EDUCATION 


70,00 


6 


LANGUAGE 


51.25 


GENERALITY 


RANK 


PROGRAM 


UTILITY 


1 


ENGLISH 


' 98,75 


2 


SOCIAL STUDIES 


76. R7 


3 


MATH 


75.00 


A 


SCIENCE 


55.00 


5 


BUSINESS EDUCATION 


50. on 


6 


LANGUAGE 


49,38 




ASSIMILATION 


RANK 


PROGRAM 


UTILITY 


1 


ENGLISH 


68.75 


2 


SOCIAL STUDIES 


71.88 


3 


MATH 


58.13 


4 


SCIENCE 


81.88 


5 


BUSINESS EDUCATION 


75.63 - 


6 


LANGUAGE 


55.00 


URGENCY 


RANK 


PROGRAM 


UTILITY 


1 


ENGLISH 


89. 3R 


2 


SOCIAL STUDIES 


70.00 


3 


MATH 




4 


SCIENCE 


71.88 


5 


BUSINESS EDUCATION 


72.50 


6 


, LANGUAGE 


50.63 


MEED 


RANK 


PRCuRAM 


UTILITY 


1 


ENGLISH 


86 .88 


2 


SOCIAL STUDIES 


73.75 


3 


MATH 


53. 13 


1 


SCIENCE 


70.63 


5 


BUSINESS EDUCATION 


62.50 


6 


LANGUAGE 


45 .00 



reOGRAM - CRITERION MATRIK 
STANDARD DEVIATION MATRIX 



RELEVANCE 



RANK 


PROGRAM 


UTILITY 


1 


ENGLISH 


29.02 


2 


SOCIAL STUDIES 




3 


MATH 


23.55 


4 


SCIENCE 


17.50 




BUSINESS EDUCATION 


19. 20 




LANGUAGE 


15 . 36 


GENERAllTY 


RANK 


PROGRAM 


UTILITY 


1 


ENGLISH 


3.31 


2 


SOCIAL STUDIES- 


15.40 


3 


MATH 


25.00 


4 


SCIENCE 


30.21 


5 


BUSINESS EDUCATION 




6 


LANGUAGE 


20.07 




ASSIMILATION 


RANK 


PROGRAM 


UTILITY 


1 


• ENGLISH 


20.27 


2 


SOCIAL STUDIES 


19.52 


3 


MATH 


25.61 


4 


.qcTFMrR 


26.92 


5 


BUSINESS EDUCATION 


22.97 


6 


LANGUAGE 


19,20 


URGENCY 


RANK 


PROGRAM 


UTILITY 


1 


ENGLISH 


11.84 


2 


SOCIAL STUDIES 


17.85 


3 




25 .98 


4 


SCIENCE 


29.99 


5 


BUSINESS EDUCATION 


15.41 


6 


LANGUAGE 


19.44 


■ NEED 


RANK 


PROGRAM 


UTILITY 


1 


ENGLSIH 


16.76 


2 


SOCIAL STUDIES 


16.54 


3 


MATH 


22.00 


4 




30,56 


5 


BUSINESS EDUCATION 


■ 21.51 


6 


LAJMGUAGE 


15.2] 



F9 



ERIC 



ASSISTANT PRINCIPAL (SCIENCE) BUDGET 



We viill assume for the purpose of this exercise that the following 
criteria; relevancej generality^ assiriilationj urgency and need, are 
to be ev'J.i>.'atf34 a^a^.nst the rnajor objectives of the school district* 



DEFBIITION; 
Relevance 

Generality 

Assimilation 

Urgency 

Need 
Utility 



The degree to vihich a program satisfies the serial and 
cultural goals and objectives of the institution. 

The degree to vihich the outputs of a given program are 
utilized by the other programs. 

The degree to which a prograia utilizes the outputs of 
all other prograriis* 

The degree to which an evaluator wishes to give explicit 
recognition to press'ore groups in the budgeting process 
with respect to a given prograra. 

The degree to which an evaluator wishes to emphasise a 
higher priority because of special circumstances. 

A real valued number that expresses an evaluator* s 
judgement of the degree to which a given program has 
the potential to contribute to the satisfaction of 
some stated criterion or objective. 



CRITSHIA EVAIUATIQT EXILRGI5KS : 
1» Rc.nk order the criteria. 



2. Assign a number betv/een 0 and 100 to each criterion^ go that these 
nuro.bsrs are consistent with your rank order. The sum total not to 
exceed 100. 



3* Check for inconsistencies between your judgments and the numerical 
representatives of your judgment, ' 



RAi^JK 


CRITERIA 


UTILITY 


1 


GENERALITY 


35 


2 


ASSIMILATION^ 


30 


3 


RELEVANCE 


20 


h 


NEED 


10 


5 


URGENCY 


5 



1. 



Rank order 
Criteria 



Need 

Generality 
Relevance 
Assimilation 
Urgency 



ho 

30 
15 
10 

5 



2* Assign Yalues 
Consistent vjith 
rank order 

3. Check for in- 
consistency. 



FIO 



CTRS^XCULUM CHAIR:.TAiN (S.S.) 



We wiU aasTJiae for the puaTpcse of this exercise that the folloiiing 
criteria^ relevance, generality, assimilation, urgency and need, are 
to be evaluated against the icajor objectives of the school district. 



DEFBIITIOI^: 

Relevance The degree to which a program satisfies the social and 

cultural goals and objectives of the institution. 

Generality The degree to which the outputs of a given program are 

utilized by the other programs * 

Assimilation The degree to which a prograra utilizes the outputs of 

*lL1 other prograins. 

Urgency The degree to which an evaluator wishes to give explicit 

recognition to pressure groups in the budgeting process 
with respect to a given program. 

Meed The degree to which an evaluator wishes to enjphasize a 

higher priority because of special circumstances • 

Utility A real valued nuniber that expresses an evaluator' s 

judgcent of the degree to which a given program has 
the potential to contribute to the satisfaction of 
soTie stated criterion or objective. 



CRITERIA E7ALUATIQI EXSRCISES ; 
1. Rank order the criteria. 



2. Assign a number betvieen 0 and 100 to each criterion, so that these 
nui-ibars are consistent ;7ith your rank order» The sum total not to 
exceed ICO. 

3- Check for inconsistencies betireen your judgments and the numerical 
representatives of your judgment. 



RiU1K 


CRITERIA 


UTILITY 


1 


GENERALITY 


35 


2 


ASSIMILATION 


30 


3 


NEED 


20 


h 


RELEVANCE 


10 




URGENCY 


5 



EXAI'IPLE: 



1. Rank order TJeed hO 

Criteria Generality 30 

Relevance 15 

Assiniilation 10 

Urgency 5 



2. Assign Values 
Consistent vrLth 
rank order 

3. Check for in- 
consistency. 



CURRICULL-.l Ca-MR:>UN (SCI.) 



lie vrifl.l^^m:Q for the purpose of this oxercise that the folloirlrg 
criteria; relevance, generality, assinilation, urgency and need, are 
to evaluated against the r^jor objectives of the school di.strict# 



DEHDIITia>:t 
Relevance 

Generality 

AsGiirdlation 

Urgency 

Need 
Utility 



The degree to uhich a program satisfies the social and 
cult'jjral goals and objectives of the institution* 

The degree to which the outputs of a given progra^i arc 
utilized by the other prograiiis. 

The degree to which a prograni utilizes the outputs of 
all other progra-xS. 

The degree to which an evaluator wishes to give explicit 
recognition to pressure groups in the budgeting process 
\n.th respect to a given program. 

The degree to which an evaluator wishes to emphasize a 
higher priority because of special circumstances. 

A real valued number that expresses an evaluator' s 
judgment of the degree to which a given program has 
the potential to contribute to the' satisfaction of 
soir.e stated criterion or objective. 

CRI7i:ilI:V EVALUATIQl EXEBCISF.S : 

1. Rank order the criteria. 

2. Assign a nunber betvjeen 0 and ICO to each criterion, ?o that these 
numbers are consistent with your rank order* The s\ni cotal not to 
exceed 100. 

3» Check for inconsistencies bettreen your judgments and "ie numerical 
representatives of your judgment • 



RAilK 


CfiKERIA 


UTILITY 


1 


NEKD 


35 


2 


RELEVANCE 


30 


3 


URGENCY 


20 


k 


GENERALITY 


10 


5 


ASSIMILATION 


5 



Rar^. order Need l;0 2. Assign Val/es 

Criteria Generality 30 Consistent with 

Relevar.ce 15 ""*-^^nl': order 

Assinilation 10 

Urgency 5 3. Check far in- ri2 



CURRICULUM CHAIRNUN (M) 



We will as suae for the purpose of this exercise that the folloifing 
criteria; relevance, generality, assimilation, urgency and need, are 
to be evaluated against the inajor objectives of the school district. 



DEFPIITION: 
Relevance 

Generality 

Assitrilation 

Urgency 

Need 
Utility 



The degree to which a program satisfies the social and 
cultural goals and objectives of the institution* 

The degreQ to which the outputs of a givon program are 
utilized by the other programs. 

The degree to which a program utilizes the outputs of 
all other programs. 

The degree to which an evaluator wishes to give explicit 
recognition to pressure groups in the budgeting process 
vith respect to a given program. 

The degree to which an evaluator wishes to emphasize a 
higher priority because of special circumstances. 

A real valued number that expresses an evaluator' s 
judgment of the degree to which a given program has 
the potential to contribute to the satisfaction of 
some stated cri'^erlon or objective. 



CRITERIA EVALUATIG; 5XERCISKS ; 

1. Rank order the criteria. 

2. Assign a nunber between 0 and 100 to each criterion, so that these 
nurr.bars are consistent with your rank order. The sum total not to 
exceed 100. 

3. Check for inconsistencies betx^een your judgnents and the numerical 
representatives of your judgmeDvt. 



1. Rank order 
Criteria 



RAIJK 


CRITEillA 


UTILITY 


1 


RELEVANCE 


35 


2 


GENERALITY 


25 


3 


ASSIMILATION 


25 


U 


URGENCY 


10 


5 


NEED 


5 



Need 

Generality 

Relevance 

Assanilation 



ho 

30 
10 



2. AssigTi Values 
Consistent vrith 
rank order 

3. O' - ^ 



COOHDINATOa (F.L.) 



V/e vdll assune for the purpose of this exercise that the follorring 
criteria; relevance, generality, assinilation, urgency and need, are 
to be evaluated against the rajor objectives of the school district. 

DEFHIITIQI: 



Relevarxe 
Generality 
ASoiinilation 
Ur gency 

Heed 



Utility 



The degree to vjhich a progran satisfies the social and 
cultural goals and objectives of the institution. 

The degree to vrhich the outputs of a given program are 
u\iliaed by the othar programs. 

The degree to vrhich a prograjn utilizes the outputs of 
all other prograiTS. 

The degree to which an evaluator ;ri.shes to give explicit 
recognition to pressure groups in the budgeting process 
v/ith respect to a given program. 

The degree to which an evaluator wishes to eicphasize a 
higher priority because of special circumstances. 

A real valued number that expresses an evaluator* s 
judgT.ent of the degree to which a given program has 
the potential to contribute to the satisfaction of 
scjT.e stated criterion or objective. 



CRI TZniA EVAIUATIQ? SmCISKS : 
1. Rank order the criteria. 



2. Assign a nunber between 0 and ICO to each criterion^ go that these 
nuT.bors are consistent vrLth your rank order. The sm, total not to 
exceed ICO. 

3. Check for inconsistencies between your judgfTients and the niunerical 
representatives of your judgment. 





CRITERIA 


UTILITY 


1 


NEED 


40 


2 


RELEVANCE 


30 


3 


UilGENCY 


15 


h 


GENERA LITV 


ic 


5 


ASSIMILATION 


5 



EXAr^LE: 

1. Rank order Need UO 2. Assign Values 

Criteria Generality 30 Conrlatont xiith 

Relevanca 15 rank order 

Asr^iriT T^-'O' 10 



CURRICULUM CHAIRMAN (ESQ.) 



We will assume for tho purpose of this exercise that the following 
criteria; relevance, gen'^ralitj'', assimilation, urgency and need, are 
to be evaluated against the major objectives of the school district. 

DEFDflTlGM; 



Relevance The degree to x^hich a program satisfies the social and 

cultural goals and objectives of the institution. 

Generality The degree to which the outputs of a given program are 

utilized by the other programs. 

Assimilation The degree to which a program utilizes the outputs of 

•11 other programs. 

Urgency The degree to which an evaluator wishe?? to give explicit 

recognition to pressure groups in the budgeting process 
w3.th respect to a given program. 

Need The degree to which an evaluator wishes to emphasize a 

higher priority because of special circumstances. 

Utility A real valued number that expresses an evaluator' s 

judgment of the degree to which a given program has 
the potential to contribute to the satisfaction of 
some stated criterion or objective. 

CRITETIIA EVALUATICK EmC ISES : 



1» Rank order the criteria. 



2. Assign a number betvjeen 0 and 100 to each criterion, so that these 
nunbsrs are consistent with your rank ordsr* The sxm total not to 
exceed ICO. 

3. Check for inconsistencies between your judgments and the numerical 
reprecentatives of your judgment. 



RAiJK 


CRITERIA 


UTILITY 


1 


GENER/XITY 


35 


2 


RELEVA^'CE 


30 


3 


XEED 


15 i 


h 


A.SSIMILATION 




5 


URGENCY 


5 



e:u:-!?ls ; 

1. Ran:< order Need UO Asr:ign V^^ 



COORDINATOR (B.E.) 



V/e vdll assune for the purpose of this exercise that the following 
criteria; relevance^ generality, assiiailaition, urgency and need, are 
to be evaluated against the rr^jor objectives of the school district. 



DEFDIITICJi 
Relevance 

Generality 

Assimilation 

Urgency 

ileed 
Utility 



The degree to which a progran satisfies the social and 
cultural goals and objectives of the institution. 

The degree to which the outputs of a given prograin are 
utilised by the other programs. 

The degree to which a prograiTi utilizes the outputs of 
all other programs. 

The degree to vihich an evaluator wishes to give explicit 
recognition to pressure groups in the budgeting process 
with respect to a given program. 

The degree to which an evaluator wishes to emphasize a 
higher priority because of special circumstances. 

A real valued nmber that expresses an evaluator^ s 
judgment of t}ie degree to which a given program has 
the potential to *;ontribute to the satisfaction of 
some stated criterion or objective. 



CRITERIA EVALUATiai EXEHOISES ; 

1. Rank order the criteria. 

2. Assign a nunbsr between 0 and 100 to each criterion, so that these 
numbars are consistent with your rank order. The sm total not to 
exceed ICO. 

3* Check for inconsistencies between your judgnents and the numerical 
representatives of your judij^ent. 



RAi-IK 




UTILITI 


1 


RELEVANCE 


60 


2 


NEED 


15 


3 


GENERALITY 


10 


h 


ASSIMILATION 


lO 


5 


URGENCY 


5 



COMPOSITE 

We v/ill assmne for the purpcse of this exercise that the follox-ring 
criteriaj relevance, generality, assimilation, urgency and need, are 
to "Se evaluated against the J^ajor objectives of the school district* 

DEFDIITIOIv]; 

Relevance The degree to ijhich a program satisfies the social and 

cult\2ral goals and objectj.ves of the in titution^ 

Generality The degree to which the outputs of a given program are 

utilized by the other programs. 

Assimilation The degree to which a program utilizes the outputs of 

all other program^. 

Urgency Tlie degree to which an evaluator wishes to give explicit 

. recognition to pressure groups in the budgeting process 
vith respect to a given program. 

Need The degree to which an evaluator Td.shes to emphasize a 

higher priority because of special circumstances. 

Utility A real valued number that expresses an evaluator' s 

Judgment of the degree to which a g,iven program has 
the potential to contribute to the satisfaction of 
seme stated criterion or objective, 

ORITZTIIA EVAIUATIQI EXIRGISES : 

!• Rank order the criteria. 

2. Assign a number between 0 and 100 to each criterion^ so that these 
nurr.bcrs are consistent iri-th, your rank order. The sum total not to 
exceed iCO. 

3. Check for inconsistencies betvmen your judgnents and the numerical 
representatives of your judgment. 





CHITERIA 


UTILITI 


1 


RELEVANCE 


27. 


7 


28 


2 


GENERALITY 


22. 


7 


23 


3 


NEED 


22. 


2 


22 


h 


ASSIMILATION 


15. 


5 


15 


5 


UKGE2ICY 


11. 


6 


12 



1. Rank order Tleed 

Criteila Genorality 
Relevance 
Assimilation 
Urgency 

:;or.sistency# 



ho 2, Assign Values 

30 ' Consistent with 

1? rank order 

10 

$ 3. Check for in- 



